Wearable computing device with electrophysiological sensors

ABSTRACT

A wearable computing device with bio-signal sensors and a feedback module provides an interactive mediated reality (“VR”) environment for a user. The bio-signal sensors receive bio-signal data (for example, brainwaves) from the user and include bio-signal sensors embedded in a display isolator, having a deformable surface, and having an electrode extendable to contact the user&#39;s skin. The wearable computing device further includes a processor to: present content in the VR environment via the feedback module; receive bio-signal data of the user from the bio-signal sensor; process the bio-signal data to determine user states of the user, including brain states, using a user profile; modify a parameter of the content in the VR environment in response to the user states of the user. The user receives feedback indicating the modification of the content via the feedback module.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority from U.S. Provisional PatentApplication No. 62/512,555 filed on May 30, 2017, and U.S. ProvisionalPatent Application No. 62/613,492 filed on Jan. 4, 2018, the contents ofwhich are hereby incorporated by reference.

FIELD

The present invention relates to wearable devices. This inventionrelates more particularly to sensors for wearable devices and wearabledevices with brain sensors. Even more particularly, this inventionrelates to wearable devices with brain sensors and methods for use inmediated reality environments.

BACKGROUND

A user may interact with a computing device for example using akeyboard, mouse, track pad, touch screen, or motion-capture devices. Asthe ways in which humans interact with computing devices change,computers may become usable for new purposes, or more efficient inperforming existing tasks. A user command to a computing device that mayrequire several commands on a keyboard may be instead associated with asingle hand gesture captured and processed by a motion-capture inputdevice. As the human body has many parts which may be controlled throughvoluntary movement, there are opportunities for capturing andinterpreting other movements for interacting with a computing device.

Bio-signals are signals that are generated by biological beings that canbe measured and monitored. Electroencephalographs, galvanometers, andelectrocardiographs are examples of devices that are used to measure andmonitor bio-signals generated by humans.

A human brain generates bio-signals such as electrical patterns, whichmay be measured/monitored using an electroencephalogram (“EEG”). Theseelectrical patterns, or brainwaves, are measurable by devices such as anEEG. Typically, an EEG will measure brainwaves in an analog form. Then,these brainwaves may be analyzed either in their original analog form orin a digital form after an analog to digital conversion.

Measuring and analyzing bio-signals such as brainwave patterns can havea variety of practical applications. For example, brain computerinterfaces (“BCI”) allow users to control devices and computers usingbrainwave signals.

SUMMARY

In accordance with an aspect of the present invention, there is provideda mediated reality device comprising: an input device and a wearablecomputing device with a bio-signal sensor, a display to provide aninteractive mediated reality environment for a user, and a displayisolator, the bio-signal sensor receives bio-signal data from the user,the bio-signal sensor comprising a brainwave sensor, wherein thebio-signal sensor is embedded in the display isolator, wherein thebio-signal sensor includes a soft, deformable user-contacting surface.

In accordance with an aspect of the present invention, there is provideda mediated reality device comprising: an input device and a wearablecomputing device with a bio-signal sensor, at least one feedback moduleto provide an interactive mediated reality environment for a user, and acontact adjuster for adjusting contact between the bio-signal sensor andthe user, the bio-signal sensor receives bio-signal data from the user,the bio-signal sensor comprising a brainwave sensor.

In accordance with an aspect of the present invention, there is provideda mediated reality device comprising: an input device and a wearablecomputing device with a bio-signal sensor, at least one feedback moduleto provide an interactive mediated reality environment for a user, and aconduction medium applicator for applying a conduction medium to a usercontacting surface of the bio-signal sensor, the bio-signal sensorreceives bio-signal data from the user, the bio-signal sensor comprisinga brainwave sensor.

In accordance with an aspect of the present invention, there is provideda mediated reality device comprising: an input device and a wearablecomputing device with a bio-signal sensor, at least one feedback moduleto provide an interactive mediated reality environment for a user, and aconduction medium applicator for applying a conduction medium to a usercontacting surface of the bio-signal sensor, the bio-signal sensorreceives bio-signal data from the user, the bio-signal sensor comprisinga brainwave sensor; the computing device having or in communication witha processor configured to: as part of the interactive mediated realityenvironment, present content via the at least one feedback module;receive user manual inputs from the input device for creating an objectin the interactive mediated reality environment; receive the bio-signaldata of the user from the bio-signal sensor; process the bio-signal datato determine user states of the user, including brain states, the userstates are processed using a user profile stored in a data storagedevice accessible by the processor and the user states include brainstates; modifying a property of the object according to the bio-signaldata of the user.

In accordance with an aspect of the present invention, there is provideda mediated reality apparatus comprising: a wearable computing devicewith a bio-signal sensor and at least one feedback module to provide aninteractive mediated reality (“VR”) environment for a user, thebio-signal sensor receives bio-signal data from the user, the bio-signalsensor comprising a brainwave sensor; the computing device having or incommunication with a processor configured to: as part of the interactiveVR environment, present content via the at least one feedback module,the content including an object in the VR environment; receive thebio-signal data of the user from the bio-signal sensor; process thebio-signal data to determine user states of the user, including brainstates, the user states are processed using a user profile stored in adata storage device accessible by the processor and the user statesincluding brain states; modify a parameter of the object in theinteractive VR environment in response to the user states of the user,wherein the user receives feedback indicating the modification of theobject via the at least one feedback module.

In some embodiments, the processor is further configured to detect theuser's interest in the object, and the parameter of the object ismodified in response to the user's interest.

In some embodiments, the processor is configured to connect with aremote feedback device for presenting an indication of the user'sinterest to an observer.

In some embodiments, another object in the VR environment is created,modified, or both in response to the user's interest in the object. Insome embodiments, the other object is an avatar of the user.

In some embodiments, the user profile includes a threshold for detectionof a virtual event presented to the user by the at least one feedbackmodule determined using the bio-signal data obtained concurrently withprevious virtual events presented to the user. In some embodiments, thethreshold for detection is modified based on the bio-signal dataobtained during the presentation of the virtual event. In someembodiments, the content being presented in the VR environment ismodified based on the threshold for detection for optimizing userengagement.

In some embodiments, the mediated reality apparatus includes a trackerfor detecting the user's physical environment and the content of the VRenvironment is modified based on properties of the physical environment.

In some embodiments, the processor communicates with effectors in theuser's physical environment for modifying the physical environment.

In accordance with an aspect of the embodiments described herein, thereis provided a bio-signal sensor including a body, an electrodeextendable into the body, the electrode having a contact end configuredto receive an electrical bio-signal from a user's skin, wherein inresponse to a downward force acting on the bio-signal sensor to urge thebio-signal sensor against the user's skin and upon contact with theuser's skin, the electrode is configured for movement into the bodyalong a movement axis, an actuator attached to the body and operativelyconnected to the electrode urging the electrode out of the body alongthe movement axis toward an extended position, wherein in the absence ofthe downward force, the electrode is disposed in the extended position,and a contact adjuster connected to the electrode, the contact adjusterincludes a handle manipulatable by the user to reduce noise theelectrical bio-signal caused by impedance of the user's hair.

According to an aspect, there is provided a mediated reality devicecomprising: a wearable computing device with a bio-signal sensor toreceive bio-signal data from a user, a display to provide an interactivemediated reality environment for the user, and a display isolator, thebio-signal sensor comprising a brainwave sensor, wherein the bio-signalsensor is embedded in the display isolator, the bio-signal sensor havinga soft, deformable user-contacting surface.

In some embodiments, the bio-signal sensor comprises a conductivecoating.

In some embodiments, the conductive coating comprises conductive ink.

In some embodiments, the mediated reality device further comprises aplurality of bio-signal sensors distributed along the display isolatorand spaced to minimize salt bridging effects.

In some embodiments, the mediated reality device further comprises anoptical device mounted on the display isolator.

In some embodiments, the bio-signal sensor comprises a cell forconductive fluid.

In some embodiments, the mediated reality device further comprises aconductive fluid reservoir in fluid connection with the cell forsupplying the conductive fluid.

In some embodiments, the display isolator comprises a sensor to measureface movement or expression.

In some embodiments, the mediated reality device further comprises aneye tracker for gaze tracking.

In some embodiments, the mediated reality device further comprises abreath sensor attached to a deformable armature.

In some embodiments, the mediated reality device further comprises anear piece having an additional bio-signal sensor.

In some embodiments, the mediated reality device further comprises acontact adjuster for adjusting contact between the bio-signal sensor andthe user,

In some embodiments, the mediated reality device further comprises atleast one feedback module to update the interactive mediated realityenvironment for a user based on bio-signal data from the user receivedat the bio-signal sensor.

In some embodiments, the computing device is in communication with aprocessor configured to: as part of the interactive mediated realityenvironment, present content via the at least one feedback module;receive user manual inputs from the input device for creating an objectin the interactive mediated reality environment; receive the bio-signaldata of the user from the bio-signal sensor; process the bio-signal datato determine user states of the user, including brain states, the userstates processed using a user profile stored in a data storage deviceaccessible by the processor and the user states including brain states;modify a property of the object according to the bio-signal data of theuser to update the interactive mediated reality environment.

According to an aspect, there is provided a mediated reality devicecomprising: a wearable computing device with a bio-signal sensor, atleast one feedback module to provide an interactive mediated realityenvironment for a user, and a contact adjuster for adjusting contactbetween the bio-signal sensor and the user, the bio-signal sensorreceives bio-signal data from the user, the bio-signal sensor comprisinga brainwave sensor.

In some embodiments, the computing device is in communication with aprocessor configured to: as part of the interactive mediated realityenvironment, present content via the at least one feedback module;receive user manual inputs from the input device for creating an objectin the interactive mediated reality environment; receive the bio-signaldata of the user from the bio-signal sensor; process the bio-signal datato determine user states of the user, including brain states, the userstates processed using a user profile stored in a data storage deviceaccessible by the processor and the user states including brain states;modify a property of the object according to the bio-signal data of theuser to update the interactive mediated reality environment.

According to an aspect, there is provided a mediated reality devicecomprising: an input device and a wearable computing device with abio-signal sensor to receive bio-signal data from a user, at least onefeedback module to provide an interactive mediated reality environmentfor the user, and a conduction medium applicator for applying aconduction medium to a user contacting surface of the bio-signal sensor,the bio-signal sensor comprising a brainwave sensor.

In some embodiments, the computing device is in communication with aprocessor configured to: as part of the interactive mediated realityenvironment, present content via the at least one feedback module;receive user manual inputs from the input device for creating an objectin the interactive mediated reality environment; receive the bio-signaldata of the user from the bio-signal sensor; process the bio-signal datato determine user states of the user, including brain states, the userstates processed using a user profile stored in a data storage deviceaccessible by the processor and the user states including brain states;modify a property of the object according to the bio-signal data of theuser to update the interactive mediated reality environment.

In some embodiments, the mediated reality device further comprises astrap integrating the bio-signal sensor.

In some embodiments, the mediated reality device further comprises adisplay isolator, wherein the bio-signal sensor is embedded in thedisplay isolator, wherein the bio-signal sensor has a soft, deformableuser-contacting surface.

According to an aspect, there is provided a mediated reality apparatuscomprising: a wearable computing device with a bio-signal sensor, toreceive bio-signal data from a user, and at least one feedback module toprovide an interactive mediated reality (“VR”) environment for the user,the bio-signal sensor comprising a brainwave sensor; the computingdevice in communication with a processor configured to: as part of theinteractive VR environment, present content via the at least onefeedback module, the content including an object in the VR environment;receive the bio-signal data of the user from the bio-signal sensor;process the bio-signal data to determine user states of the user,including brain states, the user states processed using a user profilestored in a data storage device accessible by the processor and the userstates including brain states; modify a parameter of the object in theinteractive VR environment in response to the user states of the user,wherein the user receives feedback indicating the modification of theobject via the at least one feedback module.

In some embodiments, the wearable computing device comprises theprocessor.

In some embodiments, the processor is configured to: detect the user'sinterest in the object, and modify the parameter of the object inresponse to the user's interest.

In some embodiments, the wearable computing device comprises a displayisolator, wherein the bio-signal sensor is embedded in the displayisolator, wherein the bio-signal sensor has a soft, deformableuser-contacting surface.

In some embodiments, the processor is configured to: connect with aremote feedback device for presenting an indication of the user'sinterest to an observer.

In some embodiments, the processor is configured to: create and/ormodify another object in the VR environment in response to the user'sinterest in the object.

In some embodiments, the other object is an avatar of the user.

In some embodiments, the user profile includes a threshold for detectionof a virtual event presented to the user by the at least one feedbackmodule determined using the bio-signal data obtained concurrently withprevious virtual events presented to the user.

In some embodiments, the threshold for detection is modified based onthe bio-signal data obtained during the presentation of the virtualevent.

In some embodiments, the processor is configured to modify the contentbeing presented in the VR environment based on the threshold fordetection for optimizing user engagement.

In some embodiments, the mediated reality apparatus includes a trackerfor detecting the user's physical environment and the processor isconfigured to modify the content of the VR environment based onproperties of the physical environment.

In some embodiments, the processor communicates with effectors in theuser's physical environment for modifying the physical environment.

In some embodiments, the bio-signal sensor comprises a capacitiveelectrode.

According to an aspect, there is provided a computer-implemented methodcomprising: receiving, from a bio-signal sensor, bio-signal data of auser of multiple users in a virtual or mixed environment; determining atransient electroencephalogram response of the user, based on at leastthe bio-signal data; detecting, based at least in part on the transientelectroencephalogram response, the user's notice or attendance to achange in a transient or moving stimulus in the user's visual orauditory field in the virtual or mixed environment, and ofcharacteristics of that stimulus encoded by the timecourse of thechange; signalling to an outside observer that the user noticed orattended to the stimulus; signalling, to another observer in the virtualor mixed environment, that the user noticed or attended to the stimulus;and signalling, via an event in the virtual or mixed environment, whichof the multiple users in said virtual or mixed reality environmentnoticed or attended to the stimulus.

In some embodiments, the signalling to another observer is effected viaa change of facial expression on a virtual or holographic avatar, or acolour change of said avatar.

In some embodiments, the method further comprises measuring, using inputof electrodes on the user's face or forehead, muscle activity associatedwith a facial expression of emotion; combining the user's brainwaveswith bio-signal information about the facial expression; and producing achange in state of the user's avatar in said virtual or mixedenvironment based at least in part on the combined user's brainwaves andbio-signal information.

In some embodiments, the method further comprises: detecting diminutionof the user's evoked brain response to a visual or auditory event in thevirtual or environment after repeated stimulus presentations to predicthow frequently a new stimulus of a certain type should be presented tothe user to achieve familiarity.

In some embodiments, the method further comprises: detecting diminutionof the user's evoked brain response to a visual or auditory event in thevirtual or mixed environment after repeated stimulus presentations topredict how frequently a new stimulus of a certain type should bepresented to the user to maintain a specific state of vigilance orresponsiveness, or of interest.

According to an aspect, there is provided a mediated reality devicecomprising: a wearable computing device with a bio-signal sensor, atleast one feedback module to provide an interactive mediated realityenvironment for a user, the bio-signal sensor receives bio-signal datafrom the user, the bio-signal sensor comprising a brainwave sensor,wherein the bio-signal sensor comprises: a body, an electrode extendableinto the body, the electrode having a contact end configured to receivean electrical bio-signal from a user's skin, wherein in response to adownward force acting on the bio-signal sensor to urge the bio-signalsensor against the user's skin and upon contact with the user's skin,the electrode is configured for movement into the body along a movementaxis, an actuator attached to the body and operatively connected to theelectrode urging the electrode out of the body along the movement axistoward an extended position, wherein in the absence of the downwardforce, the electrode is disposed in the extended position, and a contactadjuster connected to the electrode, the contact adjuster including ahandle manipulatable by the user to reduce noise the electricalbio-signal caused by impedance of the user's hair; wherein the computingdevice is in communication with a processor configured to: as part ofthe interactive mediated reality environment, present content via the atleast one feedback module; receive user manual inputs from the inputdevice for creating an object in the interactive mediated realityenvironment; receive the bio-signal data of the user from the bio-signalsensor; process the bio-signal data to determine user states of theuser, including brain states, the user states processed using a userprofile stored in a data storage device accessible by the processor andthe user states including brain states; modify a property of the objectaccording to the bio-signal data of the user to update the interactivemediated reality environment.

In some embodiments, the contact adjuster is configured to rotate theelectrode along a plane that is substantially perpendicular to themovement axis.

In some embodiments, the actuator includes a coil spring fixed on oneend to the body and biased against the electrode on the other end, andwherein the contact adjuster includes a shaft extending through thecompressive axis of the coil spring for translating rotational forcesperpendicular to the movement direction from the handle to theelectrode, translational forces along the movement direction from thehandle to the electrode, or both.

In some embodiments, the mediated reality device further comprises arotational limiter for limiting the rotational movement of theelectrode.

In some embodiments, the contact end of the electrode includes acollection plate and a plurality of prongs extending from the collectionplate, wherein each prong includes a distal tip for contacting theuser's skin.

In some embodiments, the radius of the distal tip is about 0.5 mm.

In some embodiments, the plurality of prongs are arranged with a prongdensity of about 15 to about 40 prongs per square centimeter.

In some embodiments, the actuator includes a plurality of actuatorscorresponding to the plurality of prongs.

In some embodiments, the contact end of the electrode has an area ofbetween about 1 cm² and about 3 cm².

In some embodiments, the extension of the electrode from the body in theextended position is adjustable using the contact adjuster.

In some embodiments, the body includes a conductive portion forreceiving the electrical bio-signal from the electrode.

In some embodiments, the conductive portion includes a conductivecoating.

In some embodiments, the conductive portion includes a conductivematerial integrated into the body.

In some embodiments, the conductive material is a carbon-loaded plastic.

In some embodiments, the body includes a spherical portion, and whereinthe sensor further comprises a housing defining a joint portionconfigured to receive the spherical portion of the body such that thebody is rotatable within the joint portion.

In some embodiments, the body includes a contact end, wherein thecontact end includes at least one groove for receiving at least aportion of the user's hair therein.

According to an aspect, there is provided a bio-signal sensorcomprising: a body, an electrode extendable into the body, the electrodehaving a contact end configured to receive an electrical bio-signal froma user's skin, wherein in response to a downward force acting on thebio-signal sensor to urge the bio-signal sensor against the user's skinand upon contact with the user's skin, the electrode is configured formovement into the body along a movement axis, an actuator attached tothe body and operatively connected to the electrode urging the electrodeout of the body along the movement axis toward an extended position,wherein in the absence of the downward force, the electrode is disposedin the extended position, and a contact adjuster connected to theelectrode, the contact adjuster including a handle manipulatable by theuser to reduce noise the electrical bio-signal caused by impedance ofthe user's hair.

In some embodiments, the contact adjuster is configured to rotate theelectrode along a plane that is substantially perpendicular to themovement axis.

In some embodiments, the actuator includes a coil spring fixed on oneend to the body and biased against the electrode on the other end, andwherein the contact adjuster includes a shaft extending through thecompressive axis of the coil spring for translating rotational forcesperpendicular to the movement direction from the handle to theelectrode, translational forces along the movement direction from thehandle to the electrode, or both.

In some embodiments, the bio-signal sensor further comprises arotational limiter for limiting the rotational movement of theelectrode.

In some embodiments, the contact end of the electrode includes acollection plate and a plurality of prongs extending from the collectionplate, wherein each prong includes a distal tip for contacting theuser's skin.

In some embodiments, the radius of the distal tip is about 0.5 mm.

In some embodiments, the plurality of prongs are arranged with a prongdensity of about 15 to about 40 prongs per square centimeter.

In some embodiments, the actuator includes a plurality of actuatorscorresponding to the plurality of prongs.

In some embodiments, the contact end of the electrode has an area ofbetween about 1 cm² and about 3 cm².

In some embodiments, the extension of the electrode from the body in theextended position is adjustable using the contact adjuster.

In some embodiments, the body includes a conductive portion forreceiving the electrical bio-signal from the electrode.

In some embodiments, the conductive portion includes a conductivecoating.

In some embodiments, the conductive portion includes a conductivematerial integrated into the body.

In some embodiments, the conductive material is a carbon-loaded plastic.

In some embodiments, the body includes a spherical portion, and thesensor further comprises a housing defining a joint portion configuredto receive the spherical portion of the body such that the body isrotatable within the joint portion.

In some embodiments, the body includes a contact end, wherein thecontact end includes at least one groove for receiving at least aportion of the user's hair therein.

In this respect, before explaining any embodiments described herein indetail, it is to be understood that the invention is not limited in itsapplication to the details of construction and to the arrangements ofthe components set forth in the following description or illustrated inthe drawings. The invention is capable of other embodiments and of beingpracticed and carried out in various ways. Also, it is to be understoodthat the phraseology and terminology employed herein are for the purposeof description and should not be regarded as limiting.

BRIEF DESCRIPTION OF DRAWINGS

Embodiments will now be described, by way of example only, withreference to the attached figures, wherein:

FIG. 1 illustrates a perspective view a wearable computing device,according to an embodiment;

FIG. 2 illustrates a rear view of the wearable computing device of FIG.1;

FIG. 3 illustrates a cross-sectional view taken along lines I-I of thewearable computing device of FIG. 2;

FIG. 4 illustrates a rear view of a wearable computing device, accordingto an embodiment;

FIG. 5 illustrates a perspective view of a portion of a face pad of thewearable computing device of FIG. 4;

FIG. 6 illustrates a cross-sectional view taken along lines II-II of theportion of the face pad of the wearable computing device of FIG. 5;

FIG. 7 illustrates a perspective view of a portion of a face pad of awearable computing device, according to an embodiment;

FIG. 8 illustrates a cross-sectional view of a face pad of a wearablecomputing device, according to an embodiment

FIG. 9 illustrates a cross-sectional view taken along lines III-Ill ofthe portion of the face pad of the wearable computing device of FIG. 7;

FIG. 10 illustrates a cross-sectional view of an electrode of a wearablecomputing device, according to an embodiment;

FIG. 11 illustrates a front view of a sound generator, according to anembodiment;

FIG. 12 illustrates a cross sectional area of the sound generator ofFIG. 11;

FIG. 13 illustrates a cross sectional area of a sound generator,according to an embodiment;

FIG. 14 illustrates a side view of a user wearing a wearable computingdevice according to an embodiment;

FIG. 15 illustrates a rear view of the user wearing the wearablecomputing device of FIG. 14;

FIG. 16 illustrates a bottom view of a scalp-contacting electrode,according to an embodiment;

FIG. 17 illustrates a side view of the electrode of FIG. 16;

FIG. 18 illustrates a side view of a scalp-contacting electrode,according to an embodiment;

FIG. 19 illustrates a side view of a scalp-contacting electrode having aconductive fluid reservoir, according to an embodiment;

FIG. 20 illustrates a side view of the scalp-contacting electrode ofFIG. 19 dispensing conductive fluid;

FIG. 21 illustrates a side view of a scalp-contacting electrode having aconductive fluid reservoir, according to an embodiment;

FIG. 22 illustrates a perspective view of the scalp-contacting electrodeof FIG. 21;

FIG. 23 illustrates a perspective view of a wearable computing device,according to an embodiment;

FIG. 24 illustrates a rear view of sensors of the wearable computingdevice of FIG. 23;

FIG. 25a illustrates a perspective view of a breath sensor, according toan embodiment;

FIG. 25b illustrates a schematic view of the breath sensor of FIG. 25ain use;

FIG. 26a illustrates a cross sectional view of a breath sensor,according to an embodiment;

FIG. 26b illustrates a perspective view of the breath sensor of FIG. 26a;

FIG. 27 illustrates a front view of a breath sensor, according to anembodiment;

FIG. 28 illustrates a cross-sectional view of an electrode, according toan embodiment;

FIG. 29 illustrates a cross-sectional view of an electrode, according toan embodiment;

FIG. 30 illustrates a non-contact electrode, according to an embodiment;

FIG. 31 illustrates a schematic representation of a memory trace,according to an embodiment;

FIG. 32 illustrates a schematic representation of a breath envelope,according to an embodiment;

FIG. 33 illustrates a schematic representation of a heartwave manifold,according to an embodiment;

FIG. 34 illustrates a schematic representation of an objectificationfield, according to an embodiment;

FIG. 35 illustrates a partial cross-sectional view of a bio-signalsensor in an uncompressed state, according to an embodiment;

FIG. 36 illustrates a partial cross-sectional view of the bio-signalsensor of FIG. 35 in a compressed state;

FIG. 37 illustrates a partial cross-sectional view of a bio-signalsensor, according to an embodiment;

FIG. 38 illustrates a perspective view of the bio-signal sensor of FIG.37;

FIG. 39 illustrates a schematic view of placement of bio-signal sensorson a user, according to an embodiment;

FIG. 40 illustrates a schematic view of placement of bio-signal sensorson a user, according to an embodiment;

FIG. 41 illustrates a perspective view of a bio-signal sensor, accordingto an embodiment;

FIG. 42 illustrates a top view of the bio-signal sensor of FIG. 41;

FIG. 43 illustrates a side view of a user wearing a wearable computingdevice having a capacitive electrode, according to an embodiment; and

FIG. 44 illustrates a partial top view of the wearable computing deviceof FIG. 43.

In the drawings, embodiments of the invention are illustrated by way ofexample. It is to be expressly understood that the description anddrawings are only for the purpose of illustration and as an aid tounderstanding, and are not intended as a definition of the limits of theinvention.

DETAILED DESCRIPTION

As used herein, the term “downward” refers to a direction toward auser's skin. Similarly, “lower” indicates a component disposed downwardrelative to another component. In contrast “upward” or “upper” are in adirection opposite the “downward” or “lower” component.

In an aspect, there is provided a computer system that is implemented byone or more computing devices. The computing devices may include one ormore client or server computers in communication with one another over anear-field, local, wireless, wired, or wide-area computer network, suchas the Internet, and at least one of the computers is configured toreceive signals from sensors worn by a user. In an implementation, thesensors include one more bio-signal sensors, such aselectroencephalogram (EEG) sensors, galvanometer sensors,electrocardiograph sensors, heart rate sensors, eye-tracking sensors,blood pressure sensors, pedometers, gyroscopes, and any other type ofsensor. The sensors may be of various types, including: electricalbio-signal sensor in electrical contact with the user's skin; capacitivebio-signal sensor in capacitive contact with the user's skin; blood flowsensor measuring properties of the user's blood flow; and wirelesscommunication sensor placed sub-dermally underneath the user's skin.Other sensor types may be possible. The sensors may be connected to awearable computing device, such as a wearable headset, wearable eyeglassframes, or headband computer worn by the user. The sensors may beconnected to the headset by wires or wirelessly. The headset may furtherbe in communication with another computing device, such as a laptop,tablet, or mobile phone such that data sensed by the headset through thesensors may be communicated to the other computing device for processingat the computing device, or at one or more computer servers, or as inputto the other computing device or to another computing device. The one ormore computer servers may include local, remote, cloud-based or softwareas a service platform (SAAS) servers. Embodiments of the system mayprovide for the collection, analysis, and association of particularbio-signal and non-bio-signal data with specific mental states for bothindividual users and user groups. The collected data, analyzed data orfunctionality of the systems and methods may be shared with others, suchas third party applications and other users. Connections between any ofthe computing devices, internal sensors (contained within the wearablecomputing device), external sensors (contained outside the wearablecomputing device), user effectors, and any servers may be encrypted.Collected and analyzed data may be used to build a user profile that isspecific to a user. The user profile data may be analyzed, such as bymachine learning algorithms, either individually or in the aggregate tofunction as a BCI, or to improve the algorithms used in the analysis.Optionally, the data, analyzed results, and functionality associatedwith the system can be shared with third party applications and otherorganizations through an API. One or more user effectors may also beprovided at the wearable computing device or other local computingdevice for providing feedback to the user, for example, to vibrate orprovide some audio or visual indication to assist the user in achievinga particular mental state, such as a meditative state.

The wearable computing device may include a camera, a display, andbio-signal measuring means to sample a user's environment as well as theuser's bio-signals, determining the user's state and context throughsensors and user input. The wearable computing device may include atleast one user-facing camera to track eye movement. In a particularaspect of the invention, the wearable computing device may be in a formresembling eyeglasses wearable on the user's face. Optionally, at leastone camera may be oriented to generally align with the user's field ofview.

In another aspect of the invention, the wearable computing deviceincludes at least one sensor adapted to being placed at or adhered tothe user's head or face. Each sensor may optionally communicate with oneanother either through wires or wirelessly. Each sensor may optionallycommunicate with a controller device either through wires or wirelessly.The controller device may be mounted to the wearable computing device inorder to reside at or near the user's head or face. Alternatively, thecontroller device may be located elsewhere on the user's body, such asin a bag or pocket of the user's clothing. The controller device mayalso be disposed somewhere outside the user's body. For example, thesensors may monitor the user, storing data in local storage mounted tothe wearable computing device, and once moving into proximity with thecontroller device, the sensors, or a transmitter of the wearablecomputing device may transmit stored data to the controller device forprocessing. In this implementation, the wearable computing device wouldbe predominantly usable by the user when located nearby the controllerdevice.

The wearable computing device may include a camera, a display andbio-signal measuring means. At least one of the bio-signal measuringmeans may employ at least one sensor in order to measure brain activity.Brain activity may be measured through electroencephalography (“EEG”)techniques electrically, or through functional near-infraredspectroscopy (“fNIRS”) techniques measuring relative changes inhemoglobin concentration through the use of near infrared lightattenuation. A sensor employing pulse oximetry techniques may also beemployed in the wearable computing device. Optionally, the wearablecomputing device may include at least one sensor measuring eye activityusing electrooculography (“EOG”) techniques. Other sensors trackingother types of eye movement may also be employed.

In various implementations, the wearable computing device may include avariety of other sensors and input means. For example, the wearablecomputing device may comprise at least one audio transducer such as asingle microphone, a microphone array, a speaker, and headphones. Thewearable computing device may comprise at least one inertial sensor formeasuring movement of the wearable computing device. The wearablecomputing device may comprise at least one touch sensor for receivingtouch input from the user.

The wearable computing device may sample from both the user'senvironment and bio-signals simultaneously or generallycontemporaneously to produce sampled data. The sampled data may beanalyzed by the wearable computing device in real-time or at a futurepredetermined time when not being worn by the user.

The wearable computing device may comprise user input detection methodsthat are adaptive and improve with use over time. Where the userattempts to command the wearable computing device, and the wearablecomputing device responds in an unexpected way, the user may attempt tocorrect the previous input by indicating that the wearable computingdevice response was incorrect, and retrying the initial command again.Over time, the wearable computing device may refine its understanding ofparticular user inputs that are corrected. Some user inputs may beeasier to successfully measure with a high degree of accuracy thanothers. It may be preferable to assign a high-accuracy input to commandthe wearable computing device that the previous input was incorrect. Forexample, tapping the wearable computing device in a particular spot mayindicate that the previous input response was incorrect. Explicittraining such as with voice recognition may also be used to configureand command the wearable computing device.

In one implementation, the wearable computing device may be in aglasses-like form factor. Glasses, with or without eyeglass elements,may be well-suited on which to mount sensors as glasses may be easilymounted to the user's face, and are easily removed. Glasses may also berelatively stable in position with respect to the user's head whenresting on parts of the user's nose and ears. In order to further reducemovement of the glasses, arm-portions of the glasses may grip sides orrear portions of the user's head. Resilient arm-portions may beparticularly useful for achieving a suitable gripping strength, therebyminimizing movement of the glasses and any sensors mounted thereupon.

Optionally, the wearable computing device may itself only providebio-signal sensors and a processor for processing measurements from thesensors. The wearable computing device may communicate thesemeasurements or data derived from processing the measurements to one ormore secondary devices, such as a Google Glass-style device. In any ofthe implementations, embodiments, or applications discussed herein, itshould be understood that some actions may be carried out by a pluralityof interconnected devices, or just one of the wearable computing devicesof the present invention. For example, the wearable computing device maynot include a display. In such an example, the wearable computing devicemay communicate visual information to the user through the use of asecond device, such as a Google Glass-style device, which does include adisplay.

Sensors usable with the wearable computing device may come in variousshapes and be made of various materials. For example, the sensors may bemade of a conductive material, including a conductive composite likerubber or conductive metal. The sensors may also be made of metal platedor coated materials such as stainless steel, silver-silver chloride, andother materials.

In addition to or instead of processing bio-signal measurements on thewearable computing device, the wearable computing device may communicatewith one or more computing devices in order to distribute, enhance, oroffload the processing of the bio-signal measurements taken or receivedby the wearable computing device. In particular, the one or morecomputing devices may maintain or have access to one or more databasesmaintaining bio-signal processing data, instructions, algorithms,associations, or any other information which may be used or leveraged inthe processing of the bio-signal measurements obtained by the wearablecomputing device. The computing devices may include one or more clientor server computers in communication with one another over a near-field,local, wireless, wired, or wide-area computer network, such as theInternet, and at least one of the computers may be configured to receivesignals from sensors of the wearable computing device.

The wearable computing device may further be in communication withanother computing device, such as a laptop, tablet, or mobile phone suchthat data sensed by the headset through the sensors may be communicatedto the other computing device for processing at the computing device, orat one or more computer servers, or as input to the other computingdevice or to another computing device. The one or more computer serversmay include local, remote, cloud-based or software as a service platform(SAAS) servers. Embodiments of the system may provide for thecollection, analysis, and association of particular bio-signal andnon-bio-signal data with specific mental states for both individualusers and user groups. The collected data, analyzed data orfunctionality of the systems and methods may be shared with others, suchas third party applications and other users. Connections between any ofthe computing devices, internal sensors (contained within the wearablecomputing device), external sensors (contained outside the wearablecomputing device), user effectors (components used to trigger a userresponse), and any servers may be encrypted. Collected and analyzed datamay be used to build a user profile that is specific to a user. The userprofile data may be analyzed, such as by machine learning algorithms,either individually or in the aggregate to function as a BCI, or toimprove the algorithms used in the analysis. Optionally, the data,analyzed results, and functionality associated with the system can beshared with third party applications and other organizations through anAPI. One or more user effectors may also be provided at the wearablecomputing device or other local computing device for providing feedbackto the user, for example, to vibrate or provide some audio or visualindication to assist the user in achieving a particular mental state,such as a meditative state.

A cloud-based implementation for processing and analyzing the sensordata may provide one or more advantages including: openness,flexibility, and extendibility; manageable centrally; reliability;scalability; being optimized for computing resources; having an abilityto aggregate information across a number of users; and ability toconnect across a number of users and find matching sub-groups ofinterest. While embodiments and implementations of the present inventionmay be discussed in particular non-limiting examples with respect to useof the cloud to implement aspects of the system platform, a localserver, a single remote server, a SAAS platform, or any other computingdevice may be used instead of the cloud.

In one implementation of the system of the present invention, aMulti-modal EEG Data-Collection and Adaptive Signal Processing System(MED-CASP System) for enabling single or multi-user mobile brainwaveapplications may be provided for enabling BCI applications. This systemplatform may be implemented as a hardware and software solution that iscomprised of an EEG headset such as the wearable computing device of thepresent invention, a client side application and a cloud servicecomponent. The client side application may be operating on a mobile ordesktop computing device. The system may provide for: estimation ofhemispheric asymmetries and thus facilitate measurements of emotionalvalence (e.g. positive vs. negative emotions); and bettersignal-to-noise ratio (SNR) for global measurements and thus improvedaccess to high-beta and gamma bands, which may be particularly importantfor analyzing cognitive tasks such as memory, learning, and perception.It has also been found that gamma bands are an important neuralcorrelate of meditation expertise.

In the same or another non-limiting exemplary implementation, possibleMED-CASP system features may include: uploading brainwaves andassociated sensor and application state data to the cloud from mobileapplication; downloading brainwave & associated data from the cloud;real-time brain-state classification to enable BCI in games or otherapplications; transmitting real-time brain-state data to other userswhen playing a game to enable multi-user games; sharing brainwave datawith other users to enable asynchronous comparisons of results; sharingbrainwave data to other organizations or third party applications andsystems; and support of cloud-based user profiles for storing personalinformation, settings and pipeline parameters that have been tuned tooptimize a specific user's experience. In this way, usage of the systemplatform can be device independent.

Each time analysis or processing of user bio-signal data (such asbrainwave data) is performed, an instance of aspects of the softwareimplementing the analysis functionality of the present invention may begenerated by the wearable computing device, initiated at either thedevice or the cloud, in order to analyze the user's private bio-signaldata using particular analysis or processing parameters applied duringthe analysis or processing. For simplicity, such an instance may bereferred to as an algorithm “pipeline”. Each instance of the pipelinemay have an associated pipeline identifier (“ID”). Each pipeline may beassociated with a particular activity type, user, bio-signal type of aparticular user, application, or any other system platform-related data.Each pipeline may maintain particular pipeline parameters determined toanalyze the user's bio-signal data in a particular way, consistenteither with previous analysis of the particular user's bio-signal data,consistent with previous analysis of one or more other user's bio-signaldata, or consistent with updated data at the cloud server derived fromnew or updated scientific research pertaining to the analysis ofbio-signal data. Pipelines and/or pipeline parameters may be saved forfuture use at the client computing device or at the cloud. When a newpipeline is created for the user, the wearable computing device or thecloud may provide a new algorithm pipeline ID to be associated with thenew pipeline at the cloud and at the device.

Each person's brainwaves are different, therefore requiring slightlydifferent tunings for each user. Each person's brain may also learn overtime, requiring the system platform to change algorithm parameters overtime in order to continue to analyze the person's brainwaves. Newparameters may be calculated based on collected data, and may form partof a user's dynamic profile (which may be called bio-signal interactionprofile). This profile may be stored in the cloud, allowing each user tomaintain a single profile across multiple computing devices. Otherfeatures of the same or another non-limiting exemplary implementationmay include: improving algorithms through machine learning applied tocollected data either on-board the client device or on the server;saving EEG data along with application state to allow a machine learningalgorithm to optimize the methods that transform the user's brainwavesinto usable control signals; sharing brainwave data with otherapplications on mobile device through a cloud services web interface;sharing brainwave data with other applications running on client devicesor other devices in the trusted network to provide for the user'sbrainwave data to control or effect other devices; integration of datafrom other devices and synchronization of events with brainwave data aidin context aware analysis as well as storage and future analysis;performing time locked stimulation and analysis to support stimulusentrainment event-related potential (“ERP”) analysis; and dataprioritization that maximizes the amount of useful informationobtainable from an incomplete data download (i.e. data is transmitted inorder of information salience). The core functionality of the MED-CASPsystem may be wrapped as an externally-usable library and API so thatanother developer may use the platform's features in the developer'sapplication(s). The library may be a static library and API for Unity3D,iOS, Android, OSX, Windows, or any other operating system platform. Thesystem platform may also be configured to use a pre-compiled algorithmsupplied by a third party within the library, including the ability fora third party developer using the library, to use the developer's ownalgorithms with the library. The system platform may also supportheadsets from a variety of vendors; personal data security throughencryption; and sharing of un-curated data (optionally usingtime-limited and fidelity limited access) through the sharing ofencryption keys.

Optionally, the wearable computing device of the present invention maybe used to implement aspects of the systems and methods described in PCTPatent Application No. PCT/CA2013/000785, filed Sep. 16, 2013, theentirety of which is incorporated by reference herein. Accordingly, thewearable computing device may be used with a computer networkimplemented system for improving the operation of one or morebiofeedback computer systems. The system may include an intelligentbio-signal processing system that is operable to: capture bio-signaldata and in addition optionally non-bio-signal data; and analyze thebio-signal data and non-bio-signal data, if any, so as to: extract oneor more features related to at least one individual interacting with thebiofeedback computer system; classify the individual based on thefeatures by establishing one or more brainwave interaction profiles forthe individual for improving the interaction of the individual with theone or more biofeedback computer systems, and initiate the storage ofthe brain wave interaction profiles to a database; and access one ormore machine learning components or processes for further improving theinteraction of the individual with the one or more biofeedback computersystems by updating automatically the brainwave interaction profilesbased on detecting one or more defined interactions between theindividual and the one or more of the biofeedback computer systems.

Optionally, the wearable computing device may be used to implementaspects of the systems and methods described in PCT Patent ApplicationNo. PCT/CA2013/001009, filed Dec. 4, 2013, the entirety of which isincorporated by reference herein. Accordingly, the wearable computingdevice may be used with a computer system or method for modulatingcontent based on a person's brainwave data, obtained by the sensors ofthe wearable apparatus of the present invention, including modifyingpresentation of digital content at at least one computing device. Thecontent may also be modulated based on a set of rules maintained by oraccessible to the computer system. The content may also be modulatedbased on user input, including through receipt of a presentation controlcommand that may be processed by the computer system of the presentinvention to modify presentation of content. Content may also be sharedwith associated brain state information.

Optionally, the wearable computing device may be used to implementaspects of the systems and methods described in PCT Patent ApplicationNo. PCT/CA2014/000004, filed Jan. 6, 2014 the entirety of which isincorporated by reference herein. Accordingly, the wearable computingdevice may be used with a computer system or method for guiding one ormore users through a brain state guidance exercise or routine, such as ameditation exercise. The system may execute at least one brain stateguidance routine comprising at least one brain state guidance objective;present at least one brain state guidance indication at the at least onecomputing device for presentation to at least one user, in accordancewith the executed at least one brain state guidance routine; receivebio-signal data of the at least one user from the at least onebio-signal sensor, at least one of the at least one bio-signal sensorcomprising at least one brainwave sensor, and the received bio-signaldata comprising at least brainwave data of the at least one user;measure performance of the at least one user relative to at least onebrain state guidance objective corresponding to the at least one brainstate guidance routine at least partly by analyzing the receivedbio-signal data; and update the presented at least one brain stateguidance indication based at least partly on the measured performance.The system may recognize, score, and reward states of meditation,thereby optionally gamifying the experience for the user. The system,using bio-signal data measurements measured by the wearable computingdevice, and in particular brainwave state measurements, may change thestate of what is displayed on the display of the wearable computingdevice. For example, in response to a determination that the user hasachieved a particular brain state, or maintained a particular brainstate for a period of time, the wearable computing device may update thedisplay to provide an indication of the determination (e.g. indicatingto the user what brain state has been achieved, and, optionally for howlong) and may further display an indication of a particular rewardassigned to the user in response to the determination.

Optionally, the wearable computing device may be used to implementaspects of the systems and methods described in PCT Patent ApplicationNo. PCT/CA2014/000256, filed Mar. 17, 2014 the entirety of which isincorporated by reference herein. Accordingly, the wearable computingdevice may implement a method including: acquiring at least onebio-signal measurement from a user using the at least one bio-signalmeasuring sensor. The at least one bio-signal measurement may include atleast one brainwave state measurement. The wearable computing device mayprocess the at least one bio-signal measurement, including at least theat least one brainwave state measurement, in accordance with a profileassociated with the user. The processing of the at least one bio-signalmeasurement includes filtering to remove line noise, transforming thesignal to an alternate domain (e.g. using Fourier or Laplacetransforms). The wearable computing device may determine acorrespondence between the processed at least one bio-signal measurementand at least one predefined device control action. In accordance withthe correspondence determination, the wearable computing device maycontrol operation of at least one component of the wearable computingdevice. Various types of bio-signals, including brainwaves, may bemeasured and used to control the device in various ways. The controllingoperation of at least one component of the wearable computing device maycomprise sharing the processed at least one brainwave state measurementwith at least one computing device over a communications network.Thresholds of brain state may be learned from each user.

Optionally, the wearable computing device may be used to implementaspects of the systems and methods described in U.S. patent applicationSer. No. 14/851,853, filed Sep. 11, 2015, the entirety of which isincorporated by reference herein. In an aspect, the wearable computingdevice may implement a method including: as part of an interactive VRenvironment, present content on the display where the content has a VRevent, desired user states, and desired effects; receive user manualinputs from an input device which have effects in the interactive VRenvironment including during the VR event; receive bio-signal data of auser from a bio-signal sensor during the VR event; process thebio-signal data to determine user states of the user, including brainstates, during the VR event, the user states are processed suing a userprofile stored in a data storage device accessible by the processor andthe user states include brain states; determine a user state score bycomparing the user states of the user to the desired user states duringthe course of the VR event; determine a performance score by comparingthe user states of the user to the desired user states during the courseof the VR event; and provide feedback to the user of the user whereinthe feedback is based on a combination of the user states score and theperformance score.

Optionally, the wearable computing device may be used to implementaspects of the systems and methods, for example, a method including: aspart of an interactive VR environment, present content via at least onefeedback module, the content including an object in the VR environment;receiving the bio-signal data of the user from a bio-signal sensor,processing the bio-signal data to determine user states of the user,including brain states, the user states processed using a user profilestored in a data storage device accessible by the a processor and theuser states including the brain states, and modifying a parameter of theobject in the interactive VR environment in response to the user statesof the user, wherein the user receives feedback indicating themodification of the object via the at least one feedback module.

In accordance with an aspect of the present invention, there is provideda wearable computing device including at least one feedback module, andat least one bio-signal sensor. The wearable computing device includesor is in communication with a processor configured, as part of amediated reality environment, to apply at least one stimulus to a uservia the at least one feedback module.

In some embodiments, the at least one stimulus provided by the at leastone feedback module affects a sensory modality including sight, sound,taste, temperature, smell, pressure or any combination thereof.

In some embodiments, stimuli from the physical, real-world environmentof a user is supplemented by the at least one stimulus from the at leastone feedback module. In such embodiments, the mediated realityenvironment is an augmented reality environment. In some embodiments, atleast one type of stimuli from the physical, real-world environment of auser is replaced by the at least one stimulus from the at least onefeedback module. In such embodiments, the mediated reality environmentis a virtual reality environment. The term “VR environment”, as usedhereinafter, refers to mediated reality environments generally, and caninclude both virtual reality and augmented reality environments. A usermay interact in the VR environment using input data such as gesturedata, manual inputs, sensor data, bio-signal sensor data, and so on.

In some embodiments, the at least one stimulus modality includes sightand the at least one feedback module includes a display. In someembodiments, the display is a stereoscopic display for displaying thevisual stimulus. The stereoscopic display optionally displays two2-dimensional images, that when observed by a user, are interpreted as asingle 3-dimensional image.

In some embodiments, the display is a head mounted display (“HMD”). Insome embodiments, the HMD includes translucent and/or transparentportions such that the displayed information is a heads-up display. Insome embodiments, the wearable computing device includes a front facingimage sensor and an image obtained from the front facing image sensor isdisplayed on the HMD for creating a virtual heads-up display.

Optionally, the wearable computing device includes a display isolatorfor reducing or eliminating visual stimuli from sources other than thedisplay. In some embodiments, the display isolator sits between theuser's face and the display. In some embodiments, the display isolatoris configured to contact the user's face. In some embodiments, thesurface of the display isolator that rests on the user's face includesthe at least one bio-signal sensor embedded thereon. In someembodiments, the portion of the display isolator that contacts theuser's face includes a soft, deformable material. In some embodiments,the display isolator defines an aperture through which a user is able toview the display. In some embodiments, the display isolator is a mask.In some embodiments, the display isolator is a shroud.

In some embodiments, the wearable computing device applies an electricalsignal for providing a feedback from the VR environment. The user mayperceive the applied electrical signal as a tingle or shock depending onthe voltage, current, and duration of the applied electrical signal.Further, the applied electrical signal may cause muscles to contract. Insome embodiments, one or more of the bio-signal sensors are configuredto apply the electrical signal such that the at least one feedbackmodule includes the one or more bio-signal sensors. The bio-signalsensors can obtain bio-signal data from the user, but when a voltage isapplied, can also apply the electrical signal. In some embodiments, theobtaining of the bio-signal data and the applying of the electricalsignal occur in half-duplex mode or in full-duplex mode. In full-duplexmode, the applying of the electrical signal may occur concurrently withthe obtaining of the bio-signal data. In full-duplex mode, the range offrequencies of the electrical signal being applied are different thanthe ranges frequencies of the bio-signal data being obtained. Thisreduces possible interference effects by the two signals. For example,the bio-signal data being obtained may have a frequency from above 0 toabout 30 Hz while the applied electrical signal has a frequency of about40 Hz or higher. In half-duplex mode, the bio-signal sensors alternatebetween applying the electrical signal and obtaining bio-signal data.The width of the pulses for applying the electrical signal and obtainingthe bio-signal data is selected to minimize the gaps in obtaining thebio-signal. In some embodiments, the width of the pulses is betweenabout 2 seconds and about 30 seconds. In some embodiments, the at leastone feedback module is an electrical signal generator for applying theelectrical signal. The electrical signal generator may be able to applya larger voltage than the bio-signal sensors. In this manner, a largerstimulus may be applied.

In some embodiments, the at least one stimulus modality includespressure and the at least one feedback modules includes a pressuretransducer. In some embodiments, the mediated reality is able to actuatethe pressure transducer such that the user is able to feel pressure,forces, vibrations or motions. In some embodiments, the pressuretransducer provides haptic feedback for the user.

In some embodiments, the at least one stimulus modality includes soundand the at least one feedback modules includes a sound generator forproviding audio stimulus to the user. In some embodiments, the soundgenerator includes two speaker drivers. One of the two speaker driversmay be placed proximate one ear of the user and the other of the twospeaker drivers may be placed proximate the other ear of the user. Thetwo speaker drivers may drive audio in stereo.

In some embodiments, the wearable computing device includes anear-mounted portion, and the ear-mounted portion includes the soundgenerator. In some embodiments, the wearable computer device includestwo ear-mounted portions, each including one or more speaker drivers.Each ear-mounted portion includes a circumaural pad. The circumaural padrests around the ear of the user. In some embodiments, the circumauralpad includes ear-adjacent bio-signal sensors. In some embodiments, theear-mounted portion includes in-ear electrodes. In-ear electrodesprovide a similar signal to scalp electrodes, but may have increasedsignal-to-noise ratios as there may be less interference from EMGsignals. In some embodiments, the at least one ear-mounted portion isdetachable from the wearable computing device. In some embodiments, theear-mounted portion includes a connector for establishing a wiredconnection that complements a receiver on a securement strap portion ofthe wearable computing device.

In using the wearable computing device, the bio-signal sensors arerequired to be in electrical connection with the user's skin in order toobtain bio-signal data. Current methods of verifying that the electricalconnection between the bio-signal sensors and the user's skin isestablished include obtaining signals from the bio-signal sensors. Theinability to obtain bio-signal data from the bio-signal sensors, ornoisy or weak bio-signal data indicates that the electrical connectionis not established or is poor. However, such processes require time tocollect and interpret the bio-signal data obtained from the sensors. Insome embodiments, the bio-signal sensors output a connection signal.When the bio-signal sensors are in electrical connection with the user'sskin, the connection signal is received by nerves on the user's skin andis perceived as a mild shock or tingle.

In order to obtain bio-signal data from a user, the bio-signal sensorsmay sit or be pressed against a user's skin. Current bio-signal sensorscan include hard metallic electrodes. When worn for an extended time,the hard metallic electrodes pressed against their skin create pressurepoints, which a user may perceive as being uncomfortable. For example,when the metallic electrodes have small contact areas against the skin,the user may perceive such electrodes as being “prickly”. In someembodiments, the bio-signal sensors include a soft, deformable materialfor distributing pressure applied by the bio-signal sensors. In someembodiments, the soft, deformable material includes a conductivecoating. In some embodiments, the conductive coating includes silver,carbon, a conductive polymer, hydrogel, UV curable conductive hydrogel.In some embodiments, the conductive polymer includespoly(3,4-ethylenedioxythiophene) (“PEDOT”). In some embodiments thePEDOT is poly(3,4-ethylenedioxythiophene)polystyrene sulfonate(PEDOT:PSS).

The conductive coating may be applied to a bio-signal sensor by dipping,screen printing, inkjet printing, spraying, or pad printing. In someembodiments, the conductive coating includes a conductive ink includingsilver, graphite or both. For example, PE872 from El DuPont de Nemoursis a silver-bearing composition that possess suitable stretchability,adhesion, and conductive properties that is compatible withpolyurethane, like thermoplastic polyurethane (TPU), and syntheticfabrics. In some embodiments, the conductive coating includes PEDOT:PSS.For example, Clevious PH1000 from Haraeus, is an aqueous PEDOT:PSSsuspension including adhesion, stretchability and conductivityadditives. In some embodiments, the PEDOT:PSS composition includesapplication additives. The application additives include surfactants,plasticizers, matting agents, solvents, binders, or combinationsthereof. For example, ionic additives to assist stretchability anelectrical conductivity is discussed in Y. Wang, C. Zhu, R. Pfattner, H.Yan, L. Jin, S. Chen, F. Molina-Lopez, F. Lissel, J. Liu, N. I. Rabiah,Z. Chen, J. W. Chung, C. Linder, M. F. Toney, B. Murmann, Z. Bao, Ahighly stretchable, transparent, and conductive polymer. Sci. Adv. 3,e1602076 (2017), which is hereby incorporated by reference.

In some embodiments, the bio-signal sensors include a conductive rubber.Conductive rubber includes conductivity additives incorporated therein.In some embodiments, bio-signal sensor include an injection moldedconductive rubber. In some embodiments, the injection molded conductiverubber includes TPU, thermoplastic elastomer (TPE), thermoplasticvulcanizate (TPV), styrene ethylene butylene streyene (SEBS) (such asLifoflex UV 60.01B03872F from HEXPOL TPE), or compression or injectionsilicones (such as ELASTOSIL® R570/60 from Wacker Chemie). In someembodiments, the conductivity additive includes silver particles, carbonparticles, carbon nanotubes, silver fibers, stainless steel fibers,PEDOT:PSS, hydrogels, or combinations thereof. In some embodiments, theconductive rubber includes adhesion additives. For example, KratonFG1901 G may be added to a SEBS rubber to increase polarity and improvecoating adhesion.

In some embodiments, the bio-signal sensor includes conductive threads.Conductive threads may be thin, flexible and durable. However, certainconductive threads may have relatively high impedance. In someembodiments, the conductive threads are used in electrodes measuringimpedance-tolerant bio-signals, such as EMG and EOG bio-signals, orwhere an operational amplifier is placed near the electrode, such aswithin one millimeter. In some embodiments, wiring of bio-sensorsinclude conductive threads providing electrical conductivity betweenelectrode regions and other electrical components. In some embodiments,the conductive thread is made entirely from metal. In some embodiments,the metallic conductive thread includes 316 stainless steel. Forexample, 316 stainless steel may be a thread spun from stainless steelfibers, such as a 8 micron fiber. In some embodiments, the metallicthread includes silver. In some embodiments, the metallic threadincludes a polymer core coated or plated with a metal. In someembodiments, the polymer core includes polyamide. In some embodiments,the conductive thread includes a conductive coating, such as PEDOT:PSS.

In some embodiments, the bio-signal sensor includes a conductive fabric.The conductive fabric may be a stretchable or non-stretchable conductivefabric. In some embodiments, the conductive fabric includes wovenconductive threads, optionally woven with non-conductive threads. Insome embodiments, the conductive fabric includes woven non-conductivethreads, optionally woven with conductive threads, and a conductivecoating applied thereon.

In some embodiments, the bio-signal sensor includes a contact electrode.The contact electrode is an object or material that is in contact withthe user's skin for the purpose of measuring electric potential orcurrent flow. In some embodiments, the bio-signal sensor includes anon-contact electrode. The non-contact electrode is an object ormaterial that is not in contact with the user's skin for measuringelectric potential through capacitive coupling. Where skin contact isnot easily achieved, for example, due to hair on a user's head, acapacitive non-contact electrode may have a better signal to noise ratiothan a contact electrode making poor or no contact with the user's head.

In some embodiments, bio-signal sensor is attached to the user with aconductive adhesive. In some embodiments, the conductive adhesiveincludes conductive ink, two-component conductive epoxy, conductivepressure sensitive adhesive, conductive transfer tape, Z-directionalconductive transfer tape.

In some embodiments, at least a portion of the bio-signal sensors areembedded in the display isolator. In some embodiments, the embeddedbio-signal sensors are level with the portion of the display isolatorthat contacts the user's face. In some embodiments, a polymer coating isapplied to the display isolator and/or the embedded bio-signal sensor tocreate a smooth surface. In some embodiments, the display isolator is asoft face pad or mask.

In some embodiments, the wearable computing device includes a contactadjuster for improving contact between bio-signal sensor and the user.The bio-signal sensors should be in contact with the skin in order toobtain accurate bio-signal data. Obstructions disposed between thebio-signal sensor and the skin may reduce the accuracy of the bio-signaldata. For example, hair disposed between a bio-signal sensor and thescalp impedes the creation of an electrical connection between thebio-signal sensor and the scalp. Further, hair may form a “mat” thatlifts the bio-signal sensor away from the skin, further impeding thecreation of an electrical connection between the bio-signal sensor andthe scalp. Current electrodes may be shaped like prongs to penetratethrough a “mat” of hair. However, such prongs may be uncomfortable whenworn. In some embodiments, the contact adjuster includes a fixationstrap. The fixation strap applies tension against the user's head. Insome embodiments, the tension applied by the fixation presses thebio-signal sensor against the skin, reducing the lift of a “mat” ofhair. In some embodiments, the contact adjuster includes a sensorhousing. The sensor housing includes a user contact surface and at leasta portion of the bio-signal sensors. In some embodiments, the sensorhousing includes a retracted position and an extended position. In theretracted position, the user contact surface is configured to contactthe user and the bio-signal sensors are flush with the user contactsurface surface (e.g. having surfaces in the same or similar plane,even) or are offset such that the bio-signal sensors are not in contactwith the user. In the extended position, the bio-signal sensors protrudefrom the user contact surface for contacting the user's skin. In someembodiments, the sensor housing defines channels through which thebio-signal sensors retract and extend. In some embodiments, a biasingmember urges the bio-signal sensor toward the extended position. In someembodiments, the user is able to manually adjust the extension of thebio-signal sensors. In some embodiments, the contact adjuster includes aplurality of extended positions and the user adjusts the bio-signalsensors into a desired extended position based on comfort and electricalcontact between the bio-signal sensors and the skin.

In some embodiments, the wearable computing device includes a conductionmedium applicator for providing a conduction medium to a skin-contactingsurface of the bio-signal sensor. The conduction medium is electricallyconductive and facilitates the electrical connection between thebio-signal sensor and the user's skin. In some embodiments, theconduction medium is a saline solution or a hydrogel. In someembodiments, the conduction medium has a viscosity of 1000-1300 cP. Insome embodiments, the conduction medium has an impedance of less than100 kΩ.

In some embodiments, the mediated reality environment includes a virtualobject interactable with the user via one or more stimulus modalities.For example, a virtual ball in a mediated reality environment may beassociated with a visual stimulus such as color, patterns, size, andrelative position of the ball; a pressure stimulus such as texture,compressibility or weight of the ball if a user “touches” or “lifts” theball in the mediated reality; an auditory stimulus, such as the soundthe ball makes as it “bounces” against ground. The one or more stimulusmodality may simulate the properties of the object in the real world, ormay be subject to properties as defined in the mediated reality.

In some embodiments, the wearable computing device includes at least oneuser input for the user to interact with the mediated realityenvironment. In some embodiments, the at least one user input includes amouse, joystick, keyboard, controller, or any combination thereof. Insome embodiments, the at least one user input includes tracking

In some embodiments, the wearable computing device includes a trackerfor measuring the position, orientation or location of the wearabledevice and the user's environment, such as 3-dimensional coordinates. Insome embodiments, the tracker includes an inertial sensor for measuringmovement of the wearable device, a gyroscope for measuring anorientation of the wearable device, an accelerometer for measuringmovement of the wearable device, a GPS for measuring a user's location,light detection and ranging (LIDAR) systems, depth cameras, beam-formingmicrophone arrays and/or other environmental detection systems, or anycombination thereof. In some embodiments, the tracker includes a gazedetector for detecting the user's gaze direction. In some embodiments,the gaze detector includes EOG sensors, an oculometer, or both.

In some embodiments, the wearable computing device includes a securementstrap for securing the wearable computing device to a user. In someembodiments, the securement strap includes bio-signal sensors integratedtherein. Securement straps are adjustable to accommodate differentusers. In some embodiments, the securement straps include elasticportions. In some embodiments, bio-signal sensors are integrated intothe securement straps. Where the securement straps are disposed below auser's hairline, the integrated bio-signal sensors would not be requiredto penetrate the “mat” of a user's hair. Accordingly, in someembodiments, the integrated bio-signal sensors include a soft,deformable contact surface. In some embodiments, to increase comfort forthe user, the soft, deformable contact surface is flush with thefixation strap.

Referring to FIG. 1 in accordance with an exemplary implementation ofembodiments described herein, there is provided a perspective view of awearable computing device 100. The wearable computing device includes ahead mounted display 110 and a face pad 120.

FIG. 2 illustrates a rear view of wearable computing device 100. Theface pad 120 includes a foam pad 121 having an exterior surface 122 andan interior matrix 126. The exterior surface 122 may be formed as partof a foam molding process, or a surface applied thereafter. In someembodiments, the interior matrix 126 includes a soft foam. In someembodiments the interior matrix 126 includes an open cell foam. Theopen-cell foam is compressible such that when the wearable computingdevice 100 is affixed to a user's head, such that the foam pad 121conforms to the user's face.

Face pad 120 may function as a display isolator for reducing oreliminating visual stimuli from sources other than head mounted display110.

In some embodiments, face pad 120 is detachably attached to wearablecomputing device 100 (e.g. face pad 120 can be attached to and detachedfrom the wearable computing device 100). In such embodiments, the facepad 120 may be a modular accessory configured to provide bio-signalsensor functionality to a VR headset.

As shown in FIG. 2, in some embodiments, face pad 120 includesbio-signal sensors disposed thereon. In some embodiments, the bio-signalsensors are electrodes 130. Electrodes 130 are distributed along theface pad 120 and may be spaced to minimize salt bridging effects. Saltbridging effects may arise, for example, due to a user's sweat or whenelectrodes are used with a conductive fluid, such as a saline solutionor hydrogel. The salt bridge forms an electrical connection betweenelectrodes and may lead to improper readings being obtained by theelectrodes. In some embodiments, the distance between electrodes 130 isat least 3 cm, preferably at least 0.5 cm.

FIG. 3 is a cross-section taken along lines I-I of face pad 120 affixedto a backbone 128 and having electrodes 130. In some embodiments, thefoam pad 121 is affixed to a backbone 128 with a conductive adhesive129. In some embodiments, the conductive adhesive 129 connects theconductive coating 124 with an exposed conductive area 125 a of aflexible printed circuit board (PCB) 125. The backbone 128 encloses theelectronics, provides structure for the face pad 120, and attaches tothe HMD, such as by velcro or other methods. In some embodiments, thebackbone 128 is made from plastic, fabric, felt, metal, or combinationsthereof, preferably plastic.

As shown in FIG. 3, in some embodiments, electrodes 130 includes aconductive coating 124. In some embodiments, a conductive coating 124 isapplied to the exterior surface 122. The coating 124 extends to the rearof the foam pad 121 to connect to a sensor manifold (not shown). Forexample, the exterior surface 122 is masked and sprayed with theconductive coating 124. In some embodiments, the conductive coating 124is PEDOT:PSS.

As shown in FIG. 4, in some embodiments, the wearable computing device100 includes an optical device 140 mounted on the foam pad 121 of facepad 120. In some embodiments, the optical device 140 is an opticalreceiver, transmitter, or optical receiver/transmitter pair. The opticaldevice 140 may be used, for example, for fNIRS brain sensing, or visiblelight measurement of blood flow and oxygenation. In some embodiments,the optical device 140 is located proximate, surrounded by, or embeddedin the electrode 130. The optical device 140 can capture additionalbio-signals for processing in conjunction with brainwave signals. Timesstamps and clock synchronization can be used, for example, the correlatemultiple signal streams.

FIG. 5 is a perspective view of a portion of foam pad 121 of wearablecomputing device 100 of FIG. 4. FIG. 6 illustrates a cross-sectionalview taken along lines II-II of the portion of foam pad 121 of wearablecomputing device 100 of FIG. 5. The optical device 140 is connected to aflexible printed circuit board (PCB) 127 through the interior matrix126. The accuracy of optical heart sensors might be improved dependingon their proximity to arteries near the surface of the user's face, suchas the facial artery and its various branches, including the lateralnasal artery and the angular artery. In some embodiments, the opticaldevice 140 is an optical heart sensor located proximate a user's nose.

As shown in FIGS. 5 and 6, in some embodiments, the electrode 130includes a soft portion 132. In some embodiments, the soft portion 132includes a closed cell foam or an elastomer. In some embodiments, theclosed cell foam is neoprene. In some embodiments, the elastomer is asoft, conductive elastomer. In some embodiments, the electrode 130 is aPEDOT:PSS coated neoprene. The PEDOT:PSS may be Clevious PH1000, dipcoated or sprayed onto the soft portion 132.

In some embodiments, for example as shown in FIG. 7, electrode 130includes detailing 136 to increase the conductivity between the frontside and backside of the electrode 130 or the adhesion to foam pad 121.In some embodiments, detailing 136 includes a hole or channel disposedthrough or partially through the electrode 130, having conductivecoating 124 disposed therethrough. FIG. 9 illustrates a cross-sectionalview taken along lines III-Ill of the portion of foam pad 121 ofwearable computing device 100 of FIG. 7. As shown, conductive adhesive129 connects the conductive element 123 with an exposed conductive area125 a of a flexible printed circuit board (PCB) 125.

FIG. 8 illustrates a cross-sectional view of a foam pad 121 of a facepad 120, according to an embodiment. In some embodiments, the electrode130 includes a conductive base 134. The conductive base 134 may be madefrom plastic, metal, or combination thereof. In some embodiments, thebase 134 is molded into soft portion 132 or attached with a conductiveadhesive.

In some embodiments, for example, as shown in FIG. 10, a fixation member137 connects electrode 130 with a flexible PCB 138, a conductive thread,or a wire. In some embodiments, the thread is a spun stainless steelthread.

Having reference to FIG. 28, an alternative electrode 130 is shown. Theelectrode includes a soft portion 132. The soft portion 132 includes anopen-cell foam that is optionally coated with a conductive layer, suchas PEDOT:PSS. The foam is soaked with a conductive fluid prior to use.In some embodiments, the conductive fluid is saline, or an electrodegel/fluid. The electrode 130 has a conductive coating, such asconductive adhesive 129 as shown in FIG. 28, or flexible PCB forelectrically connecting the soft portion with the HMD 110 or sensorelectronics. In some embodiments, the electrode 130 is attached to thestrap 111 by hook and loop connectors 135, such as Velcro.

Having reference to FIG. 29, in some embodiments, a conductive fluidreservoir 200 may be fluidly connected to the soft portion for supplyingconductive fluid 177 to the soft portion of the electrode 130. Thereservoir 200 includes a refilling port 202 for supplying the reservoir200 with conductive fluid 177. In some embodiments, an electricalconnection 204 is provided. Use of conductive fluid 177 may reduce theimpedance and may improve the connection over electrodes withoutconductive fluid.

Having reference to FIGS. 23 and 24, in some embodiments, the face pad120 includes pressure and/or strain sensors to measure face movement.The sensors augment other sensors, such as facial EMG, to determine thefacial expression the user is exhibiting. In some embodiments, thepressure and/or strain sensors are in the form of segmented facecushions 190. Facial movement 191 causes differential pressure andcompression of the segmented face cushions 190. Piezoelectric or printedstrain sensors 192 on the surface of cushion 190 for measuring strain.The sensors 192 are aligned with the muscles of the face, such as theorbicularis oculi. The bulk impedance measurement through a conductivefoam interior 193 of the cushion 190 can measure the compression of thecushion 190. In some embodiments, the surface of the cushion 190includes conductive surfaces between adjacent segments to measurepressure changes between the segments caused by lateral movement of theskin. The movement causes the impedance between the segments to vary. Insome embodiments, the cushion 190 includes a piezoelectric resistive orprinted strain sensor 192 on a bottom surface of the cushion 190 tomeasure pressure.

Facial bio-signal sensors such as electrodes 130 or sensors 192 mayfurther yield facial expression information (which may be difficult toobtain using cameras in a VR headset). Muscles specifically around theeyes play an important role in conveying emotional state. Smiles, forexample, if accompanied by engagement of the muscles at the corners ofthe eyes are interpreted as true smiles, in contrast to those that areput on voluntarily. EOG signals provide information about eye movements.Basic gaze direction and dynamic movement can be estimated in real-timeand can thus be used as a substitute for optical methods of eye trackingin many applications. In some embodiments, such information can berendered on an object in a VR environment, for example, on the eye(s) ofan avatar of the user in the VR environment. Measurement of the EOGsignal is also important for noise free interpretation of the EEGsignal. fNIRS sensors if used can provide supplemental information aboutactivity in the frontal region of the brain with high spatial accuracy.Other sensors tracking other types of eye movement may also be employed.

In some embodiments, for example as shown in FIG. 4, the wearablecomputing device 100 includes an optical eye tracker 150 for user gazetracking. In some embodiments, electrodes 130 are used to obtain EOGdata for gaze tracking.

In some embodiments, for example as shown in FIG. 4, the wearablecomputing device 100 includes a breath sensor 160. When worn, the breathsensor 160 may be located proximate a user's nose. Having reference toFIGS. 25a and 25b , breath sensor includes a turbulence inducer 162 anda pressure transducer 164 attached to a deformable armature 161. Thedeformable armature allows the user to adjust the breath sensor 160 toadjust it to an optimal position for their face. The pressure transducer164 measures the pressure vibrations from the air flow due to a user'sbreathing. In some embodiments, the pressure transducer 164 includes anelectret microphone, dynamic microphone, a piezo-electric device. Theturbulence inducer 162 causes a user's breath flowing toward sensor 160to increase in turbulence such that it can be detected by pressuretransducer 164. The breath sensor 160 may be placed under the user'snose, or to the side. When placed under the user's nose, the pressuretransducer better detects the lower frequency pressure modulations.Having reference to FIG. 26a , in some embodiments, the turbulenceinducer 162 includes a grate. The turbulence inducer vibrates when auser's breath flows past. The vibration is detected by the pressuretransducer 164. FIG. 26b illustrates a perspective view of the breathsensor 160 embodiment of FIG. 26 a.

Having reference to FIG. 27, in some embodiments, the breath sensor 160is integrated into a nose guard of a HMD and blocks stray light. In someembodiments, the turbulence inducer 162 is a series of ventilationholes.

Having reference to FIGS. 11 and 12, in some embodiments, the wearablecomputing device 100 includes a sound generator 1140. In someembodiments, the sound generator is a headphone including an armature142 housing and headphone earpiece 144. In some embodiments, the breathsensor 160 is supported by the armature 142. The earpiece 144 includes aspeaker 146 and electrodes 149. In some embodiments, the sound generator1140 includes a conductive pad 148 electrically connecting to aconductive pad 102 of the wearable computing device 100. In someembodiments, the conductive pad 102 is disposed on a strap 111 of theHMD 110. The connection of the conductive pad 148 connects theelectrodes 149 of the sound generator 1140 to the HMD 110. In someembodiments, the electrical connection is effected by the application ofmechanical pressure. In some embodiments, the sound generator 1140 isused independently or without head mounted display 110. In someembodiments, the conductive pad 148 is used as an electrode to measurebio-signals if the headphones are used independently from the HMD 110.

In some embodiments, the earpiece 144 includes a pad 145 attached to anearpiece body 147. The pad 145 includes an interior 1148, an exteriorsurface 1150, and a coupler 152 for attaching the pad 145 to theearpiece body 147. In some embodiments, the interior 1148 is anopen-cell foam. In some embodiments, the exterior surface 1150 is athermoplastic urethane or a synthetic leather, or other suitablematerial for headphone earpads. The pad 145 includes a conductivecoating 154 applied on the exterior surface 1150. The conductive coating154 is electrically connected to a conductive flange 156 on the earpiecebody. The conductive flange 156 connects to the conductive pad 148 via awire 158. In some embodiments, additional electrical connection betweenthe conductive flange 156, the conductive pad 148, and wire 158 isprovided. The pad 145 includes electrodes 149 disposed thereon and arein electrical connection with the conductive coating 154. The electrodes149 may be disposed on the pad 145 similar to how the electrodes 130 aredisposed on the foam pad 121 of the HMD 110.

Having reference to FIG. 13, in some embodiments, the sound generator1140 includes a connector 102 b that provides electrical connection to aconnector 102 a of the HMD 110. In some embodiments, the connectors 102b and 102 a include complementary 1.5 mm stereo audio connectors ormagnetic connectors. In some embodiments, a wire 159 electricallyconnects the connector 102 b and the conductive coating 154.

FIG. 14 illustrates a side view of a user 10 wearing a wearablecomputing device 100, according to an embodiment. As shown in FIG. 14,when worn, strap 111 of wearable computing device 100 fixes the HMD 110on user 10. The strap 111 includes bio-signal sensors such as electrodes170 for obtaining bio-signals from the scalp or skin 12 of user 10. Thestrap 111 optionally includes other bio-signal sensors such asnon-contact electrodes 180. FIG. 15 illustrates a rear view of user 10wearing wearable computing device 100 according to the embodiment shownin FIG. 14.

Having reference to FIGS. 16 and 17, in some embodiments, electrode 170includes electrode pins 174 attached to the strap 111 by biaser 172. Thestrap 111 defines apertures 171 sized to receive the electrode pins 174.The electrode pins 174 may be made of conductive or nonconductiveplastic with a conductive coating applied thereon. Alternatively, thepins 174 may include a replaceable hydrogel tip. The electrode pins 174are perpendicularly displaceable with respect to the strap 111. When theHMD 110 is worn, the electrode pins 174 first make contact against theuser's scalp. As the strap 111 is tightened, the electrode pins 174 moverelative to the strap 111 through the apertures 171. The biaser 172resists this movement and applies pressure keeping the electrode pins174 against the scalp. This allows the pressure to be distributed on theuser's scalp between the strap 111 and the electrode pins, as comparedto a fixed electrode pin where all pressure is applied at the electrodepin, thereby reducing the amount pressure acting on the scalp at theelectrode pins 174. In some embodiments, the biaser 172 is a deformablebase. In some embodiments, the deformable base includes an elastomerretaining the base. In some embodiments, the elastomer is a softelastomer, such as 40A durometer silicone rubber.

Having reference to FIG. 18, in some embodiments, the biaser 172includes pin guides 173, each pair of pin guides 173 attached to aspring 178. In some embodiments, the electrode pin 174 includes anadjustment portion 179 a allowing a user to manually adjust theelectrode pin 174 to move it through the user's hair to make contactwith the user's scalp, for example, by wiggling the pin 174.

FIG. 19 illustrates a side view of a scalp-contacting electrode 170having a conductive fluid reservoir 175, according to an embodiment.FIG. 20 illustrates a side view of the scalp-contacting electrode ofFIG. 19 dispensing conductive fluid 177.

As shown in FIGS. 19 and 20, in some embodiments, electrode 170 mayinclude a conductive fluid reservoir 175 containing a conductive fluid177 therein. Electrode pins 174 may be biased against skin 12 of user 10by spring 178 attached to pin guides 173. The electrode pin 174 includesa conduit 179 b for receiving conductive fluid from the reservoir 175 todistribute the conductive fluid to the tip of the of electrode pin 174.The conduit 179 b is sized depending on the viscosity of the conductivefluid. In some embodiments, the reservoir 175 includes a loading port176 for refilling the reservoir 175 with conductive fluid 177. In someembodiments, the reservoir 175 is deformable, thereby pressurizing theconductive fluid in the reservoir 175 and urges the conductive fluid 177through the conduit 179 b. In some embodiments, the reservoir 175 isfilled with a syringe or suctions conductive fluid from a conductivefluid source by first depressing the reservoir 175.

FIG. 21 illustrates a side view of a scalp-contacting electrode having aconductive fluid reservoir 175, in an embodiment. FIG. 22 illustrates aperspective view of the scalp-contacting electrode of FIG. 21. As shownin FIGS. 21 and 22, conductive fluid reservoir 175 may be attach tostrap 111 by one or more supports 2100. In some embodiments, support2100 may be deformable, and may include an elastomer. In someembodiments, the elastomer is a soft elastomer, such as 40A durometersilicone rubber.

In accordance with an aspect of the embodiments described herein, strap111 may include sensors such as bio-signal sensors 3500 for obtainingbio-signals from the scalp or skin 12 of user 10. With reference to FIG.39, there is provided a bio-signal sensor 3500. The sensor 3500 isconfigured to receive a bio-signal from a user 10, preferably, from theuser's head or through the skin 12 of user 10. With reference to FIG.40, the bio-signal sensor 3500 can be included on an apparatus 4000, forexample on a support portion 4002 such as strap 111 of wearablecomputing device 100. The apparatus 4000 optionally includes at leastone deformable portion 4004, for example, made from foam, connected tothe support portion 4002 to provide comfort and/or support when theapparatus 4000 is worn by the user 10.

With reference to FIGS. 35 and 36, the bio-signal sensor 3500 includes abody 3520, having a spherical portion 3528; an electrode 3530 extendableinto the body 3520, the electrode 3530 having a contact end 3532configured to receive an electrical bio-signal from a user's 10 skin 12,wherein in response to a downward force acting on the bio-signal sensor3500 to urge the bio-signal sensor 3500 against the user's skin 12 andupon contact with the skin 12 of user 10, the electrode 3530 isconfigured for movement into the body 3520 along a movement axis 3522;an actuator 3540 operatively connected to the electrode 3530 for urgingthe electrode 3530 out of the body 3520 along the movement axis 3522toward an extended position, wherein in the absence of the downwardforce, the electrode 3530 is disposed in the extended position; and acontact adjuster 3550 connected to the electrode 3530, the contactadjuster 3550 includes a handle 3552 manipulatable by the user to reducenoise the electrical bio-signal caused by impedance of the user's hair.

In use, a force having a downward component is applied to urge thebio-signal sensor 3500 against the skin 12 of user 10 to receive anelectrical signal from the user 10. The electrode 3530 moves along themovement axis 3522 into an electrode receiving space 3524 of body 3520from an extended position toward a retracted position (see, for example,FIG. 36). However, the user's hair may impede the ability of thebio-signal sensor 3500 to receive an electrical signal from the skin 12of user 10. For example, the user's hair may form a barrier (or “mat”)that acts as an insulation layer between the contact end and the user'sskin. The insulation layer impedes or prevents the receiving of theelectrical signal. As such, in some embodiments, the bio-signal sensor3500 is configured to reduce the impedance effects of the user's hair.

In some embodiments, the contact end 3532 of the electrode 3530 includesa collection plate 3534 and a plurality of prongs 3536 extending fromthe collection plate 3534. Each prong includes a distal tip 3537 forcontacting the skin 12 of user 10. Whereas with an electrode having asingle contact surface, the user's hair may form a mat under the singlecontact surface, an interstitial volume 3538 defined by the prongs 3536,the collection plate 3534, and the skin 12 of user 10 may receive theuser's hair and reduce or prevent the formation of a mat under thedistal tips 3537 of the prongs. In some embodiments, the extension ofthe electrode 3530 from the body 3520 in the extended position isadjustable using the contact adjuster 3550. In some embodiments, contactadjuster 3550 includes a compression fitting, or threading that mateswith the electrode or the body for adjusting the extension of theelectrode 3530 in the extended position. The extension of the electrode3530 from the body 3520 accommodates users with different volumes ofhair. For example, a user with thick, long hair, may have a relativelygreater volume of hair, which may create an electrical barrier if a matis formed. For such users, the extended position may be adjusted suchthat the electrode 3530 extends further from the body 3520 than forusers with shorter or no hair.

In some embodiments, the contact adjuster 3550 is configured to move theelectrode along the movement axis 3522. In some embodiments, the handleis configured for lifting the electrode 3530 when urged against the skin12 of user 10 and repositioning the electrode for placement against theskin 12 of user 10. In some embodiments, the movement of the contactadjuster 3550 moves the plurality of the prongs 3536 collectively. Forexample, in some embodiments, the contact adjuster 3550 is connected tothe collection plate 3534 and is configured to move the collectionplate. The movement of the collection plate 3534 causes the plurality ofprongs 3536, which extend from the collection plate 3534, to move.

On the application of a downward force, the electrode 3530 moves alongthe movement axis 3522 into the body 3520 (see FIG. 36). Where there issignificant retraction of the electrode 3530 into the body 3520, thebody 3520 may become proximal to the skin 12 of user 10. This may cause,for instance, the user's hair disposed under the body 3520 of the sensor3500 may form a barrier layer preventing good contact between theelectrode 3530 and the skin 12 of user 10. Thus, in some embodiments,the body 3520 includes a contact end 3526 including at least one groove3529 for receiving at least a portion of the user's hair therein.

In order to provide better comfort for a user, the pressure of theelectrode 3530 against the skin 12 of user 10 may not be excessive. Insome embodiments, the distal tips 3537 of the plurality of prongs 3536are rounded. In contrast to a pointed tip, a rounded tip distributes theforce applied to the skin over a greater area. In some embodiments, theradius of the distal tip is between about 0.25 mm and about 1 mm. Insome embodiments, the radius of the distal tip is about 0.5 mm. Thenumber and spacing of the prongs 3536 are selected such that thepressure applied to the skin 12 of user 10 is not excessive and hassufficient contact area to receive good adequate signal from the user'sskin while maintaining sufficient void volume between prongs 3536 toreceive the user's hair. In some embodiments, the electrode 3530 has aprong density of about 15 to 40 prongs per square centimeter. In someembodiments, the electrode 3530 has a prong density of about 25 pins persquare centimeter.

A greater area of the contact end of the electrode 3530 may providebetter electrical readings. However, when the area is too large, it maynot conform well to the skin. One reason for this is that the skin is,typically, not perfectly flat. Increased area of the contact end of theelectrode also increases the likelihood that the skin's curvature bendsaway, resulting in a loss of contact for the electrode. Thus, in someembodiments, the area of the contact end of the electrode 3530comprising the prongs 3536, including the interstitial area betweenprongs, is between about 1 cm² and about 3 cm². In some embodiments, thearea of the contact end of the electrode 3530 comprising the prongs,including the interstitial space between prongs, is about 1.5 cm². Insome embodiments, the shape of the contact end 3532 of the electrode isround or polyhedral. The shape of the contact end 3532 may help move theuser's hair to reduce or prevent the impedance effects of the user'shair.

In some embodiments, the contact adjuster 3550 is configured to rotatethe electrode along a plane that is substantially perpendicular to themovement axis. The rotational movement may move the hair disposed underthe sensor 3500. In some embodiments where the sensor includes aplurality of prongs 3536, the rotational movement may move the hair intothe interstitial volume 3538. In some embodiments, the rotationalmovement of the contact adjuster 3550 is unrestricted. In someembodiments, the rotational movement of the contact adjuster 3550 islimited.

In some embodiments, the actuator 3540 includes a spring, a piston, acompressible material, or combination thereof. In some embodiments, theactuator 3540 includes a spring 3542. In some embodiments, the spring3542 is a coil spring. The spring 3542 is disposed within the electrodereceiving space 3524 such that one end is biased against an upper end3526 of the body against the electrode 3530 such that the electrode 3530is urged away from the electrode receiving space 3524 toward theextended position. In some embodiments, the spring 3542 biases againstan upper end of the collection plate 3532 of the electrode 3530. When adownward force is applied to the sensor 3500 and when the electrode 3530is against the skin 12 of user 10, the spring 3542 resists the movementof the electrode 3530 into the body 3520 such that a force is translatedto the electrode 3530 urging it against the skin 12 of user 10.

In some embodiments, the spring 3542 is fixed on one end to the body3520 and biased against the electrode 3530 on the other end, and whereinthe contact adjuster 3550 includes a shaft 3554 extending through acompressive axis 3544 of the spring 3542 for translating rotationalforces perpendicular to the movement direction from the handle 3552 tothe electrode 3530, translational forces along the movement directionfrom the handle to the electrode, for both. In some embodiments, thecompressive axis is co-axial or substantially co-axial with the movementaxis 3522. In some embodiments where the spring 3542 is a coil spring,the coils of the coil spring are coiled around the shaft 3554 of thecontact adjuster 3550.

In some embodiments, the actuator 3540 includes a plurality of actuators(not shown) corresponding to the plurality of prongs 3536. In someembodiments, the plurality of actuators individually bias the prongsagainst the skin 12 of user 10. This may allow, for instance, betterconformity of the sensor against the skin 12 of user 10 as the skin maynot be perfectly flat.

The electrical bio-signal received by the electrode 3530 may betransmitted to a signal receiver, such as a processor or other computingdevice (not shown). In some embodiments, the signal receiver receivesthe electrical bio-signal from the body 3520 of the sensor. In someembodiments, the body includes a conductive portion 3527 for receivingthe electrical bio-signal from the electrode. The conductive portion3527 may be a conductive coating, a conductive material integrated intothe body, or both. In some embodiments, the conductive coating is aconductive paint, such as a metallic paint, or a carbon paint. In someembodiments, the metallic paint includes silver, gold, silver-silverchloride, or a combination thereof. In some embodiments, the conductivematerial is a carbon-loaded plastic, or a conductive metal. In someembodiments, the body is 3D printed with a conductive materialincorporated therein. In some embodiments, impedance between theelectrode and a connection on the sensor for a wire from the signalreceiver is less than about 1 kΩ. In some embodiments, the impedancebetween the electrode and the connection on the sensor is from about 1Ωto about 500Ω. In some embodiments, the connection is on the body 3520or on a housing 3760 of a sensor 3700 shown in FIG. 37.

In some embodiments, the actuator 3540 electrically connects theelectrode 3530 to the body 3520. For example, an electrical bio-signalmay be transmitted from the electrode 3530 to the body 3520 via theactuator 3540. In some embodiments where the actuator 3540 includes aspring 3542, the spring 3542 is conductive. For example, a spring 3542biased on one end against a collection plate 3534 and on the other endagainst the body 3520, the spring may act as a conductor.

In accordance with an aspect of the embodiments described herein, strap111 may include sensors such as bio-signal sensors 3700 for obtainingbio-signals from the scalp or skin 12 of user 10. Having reference toFIGS. 37 and 38, in some embodiments, a sensor 3700 includes a gimbal3770 configured to orient the electrode 3730 normal or substantiallynormal to the skin 12 of user 10. A normally oriented electrode 3730 mayhave better contact with the user's skin. For example, where prongs 3736are the same length, a normal orientation prevents the angular contactwith the user's skin where certain prongs are not lifted off from theuser's skin. Further, where the electrode 3730 contacts the skin at anangle, one or more of the prongs 3736 may be pushed up by the hair. Insome embodiments, body 3720 includes a spherical portion 3728, whereinthe sensor further includes a housing 3760 defining a joint portion 3762configured to receive the spherical portion 3728 of the body 3720 suchthat the gimbal 3770 includes the spherical portion 3728 and the jointportion 3762. In some embodiments, the spherical portion 3728 isremovably receivable by the joint portion 3762. In some embodiments, theinterface between the joint portion 3762 and the spherical portion 3728includes a friction reducing agent. In some embodiments, the frictionreducing agent is a carbonaceous material. In some embodiments, thecarbonaceous material is integral to at least a portion the body 3720,the housing 3760, or both. In some embodiments, the housing 3760includes an electrical connection portion for establishing an electricalconnection between the sensor 3700 and a signal receiver.

In some embodiments, body 3720 includes at least one groove 3729 forreceiving at least a portion of the user's hair therein.

In some embodiments, at least a portion of the conductive portion 3727is disposed in or on the spherical portion 3728. In some embodiments,the electrical bio-signal received from the electrode 3720 istransmitted to the housing 3760 from the body 3720. In theseembodiments, the signal received may connect to the housing 3760. Insome embodiments where a friction reducing agent is included, thefriction reducing agent includes or is a conductivity modifier toimprove impedance. In some embodiments, the conductivity modifier is ametal powder, graphite, carbon nanotubes, metal-coated glass or plasticbeads. For example, where the friction reducing agent is a carbonaceousmaterial integral to the body 3720, the carbonaceous material mayprovide both friction reduction and conductivity. In some embodiments, awire on a support portion 4002 of a head-mounted apparatus 4000 isconnected at one end to the sensor 3700.

Having reference now, to FIGS. 41 and 42, in some of the embodimentswhere the rotational movement is limited, the sensor 4100 includes arotational limiter 4170 for limiting the rotational movement of theelectrode 4130. If the hair is rotated excessively in a singledirection, the hair may become wrapped or tangled. In some embodiments,the rotational limiter allows an oscillatory movement along a rotationalaxis for the electrode to get between the user's hairs. In someembodiments, the rotational limiter limits the rotational movement to atleast about 0.25 radians. In some embodiments, the rotational limiter4170 includes a slot 4172 and a key 4174 configured to rotaterestrictively within the slot 4172. The movement of the electrode 4130with respect to the body 4120 are limited by the slot 4172 and the key4174. In some embodiments, the upper end 4126 of the body 4120 definesthe slot 4172 and the shaft 4154 of the contact adjuster 4150 includesthe key 4174. In some embodiments, the rotational limiter includes astop disposed in the body, the electrode, the shaft, or any combinationthereof. In some embodiments, a housing 4160 is configured to receivebody 4120.

In some embodiments, a light connected to the processor indicates abrain state at the sensor 3500 or sensor 3700. In some embodiments, thebrightness or color of the light is modified according to an event inthe brain, such as an event related potential, a continuous EEG, acognitive potential, a steady state evoked potential, or combinationthereof. In some embodiments, the light is integral with the sensor ormounted proximate the sensor on a support portion of a head-mountedapparatus.

Having reference to FIG. 30, in some embodiments, non-contact electrodes180 include a conductive layer 182 and a conductive noise layer 184 witha dielectric layer 186 disposed therebetween. The conductive noise layer184 reduces the noise in the signal obtained by the electrode 180. Theconductive noise layer 184 may be an active guard or a ground plane. Insome embodiments, a dielectric layer 188 is applied to a user facingside of the conductive layer 182. The conductive layer 182 connects tothe HMD 110 or sensor electronics via a wire 189.

In some embodiments, a non-contact electrode may take the form ofcapacitive electrode 4300, as shown in FIG. 43. FIG. 43 illustrates aside view of user 10 wearing a wearable computing device 100 having abio-signal sensor in the form of a capacitive electrode 4300, accordingto an embodiment. FIG. 44 illustrates a partial top view of wearablecomputing device 100 of FIG. 43.

In some embodiments, strap 111, which fixes the HMD 110 on user 10,includes one or more capacitive electrodes 4300, for example, positionedadjacent a top of the head of user 10 and the back of the head of user10, as shown in FIG. 43. Electrodes 4300 may be disposed in strap 111 ofwearable computing device 100 to receive bio-signal data of user 10. Insome embodiments, received bio-signal data may include brainwave data ofuser 10. In some embodiments, capacitive electrode 4300 may be anoncontact electrode that does not come into direct contact with skin 12of user 10.

Strap 111 may include a compressible foam 4302 which may conform to theshape of the head of user 10. In some embodiments, compressible foam4302 may be formed of an open cell foam, such as open cell foam materialknown to a user skilled in the art. Compressible foam 4302 may becompressible such that when the wearable computing device 100 is affixedto the head of user 10, compressible foam 4302 conforms to the head ofuser 10. In use, the compressible foam 4302 may be compressed andconform to the head of user 10 by clinching of strap 111 that securesHMD 110 to user 10.

In some embodiments, on a surface of compressible foam 4302 adjacentuser's 10 head, a conductive layer 4304 of capacitive electrode 4300 issecured to compressible foam 4302.

Conductive layer 4304 may have a thickness between 1 and 100 μm, in anexample 20 μm. Conductive layer 4304 may be formed of a conductivematerial such as a polymer substrate with conductive ink, a conductivepolymer, conductive fabric or a flexible PCB.

Conductive layer 4304 may be insulated adjacent the head of user 10 withan insulating layer 4306. Insulating layer 4306 forms a dielectricmedium, creating a capacitive coupling between conductive layer 4304 andskin 12 of user 10. In some embodiments, hair or other body tissue ofuser 10 may further contribute to the dielectric formed by insulatinglayer 4306 and the capacitive coupling may form across hair or otherbody tissue of user 10. Hair of user 10 may be compressed and held inplace by the pressure exerted by compressible 4302.

Insulating layer 4306 may have a thickness between 1 and 100 μm, in anexample 50 μm. Insulating layer 4306 may be formed of a polymer, forexample, polyester.

Insulating layer 4306, by providing a minimal insulating layer betweenconductive layer 4304 and skin 12 of user 10, may moderate variabilityin the capacitive coupling between conductive layer 4304 and skin 12 ofuser 10 caused by variances in the properties of user's 10 hair.Insulating layer 4306 may also minimize salt bridging effects that mayarise, for example, due to user 10 sweat creating a salt bridge formingan electrical connection between electrodes leading to improper readingsbeing obtained by the electrodes.

In some embodiments, conductive layer 4304 may be connected to the HMD110 or sensor electronics, for example, a signal conditioning andamplification circuit, via a wire (not shown).

In various implementations, the wearable device 100 may include atracker or other sensors, input devices, and output devices. In someembodiments, for example, the tracker is an inertial sensor formeasuring movement of the device 100. It detects the 3-dimensionalcoordinates of the wearable device 100 and accordingly its user'slocation, orientation or movement in the VR environment including theuser's gaze direction. The tracker, for example, comprises one or moreaccelerometers and/or gyroscopes. The wearable device 100 may comprise atouch sensor for receiving touch input from the user and tactile devicefor providing vibrational and force feedback to the user. The wearabledevice 100 may further include input devices such as mouse, keyboard andjoystick. In some embodiments, the wearable device 100 may be a trainingsystem.

Electrical signals may be measured on other regions of the head and maybe mounted to the supporting architecture of the wearable device 100.Typically these are elasticized fabric. Sensors that measure scalppotentials would typically have a fingered design to allow theconductive electrodes to reach through the hair to reach the surface ofthe scalp. The fingers may be springy to allow for comfort and allow forthe user to manipulate them in a fashion that will spread and dispersehair to facilitate a low impedance interface to skin of the scalp.Capacitive electrodes may also be used, for example, capacitiveelectrode 4300 as discussed above. Capacitive electrodes may provide fora slight air gap between the electrode and the scalp.

Many electrodes may be used if possible to allow for a higherdimensional bio-signal to facilitate denoising signal processing and toacquire more accurate spatial information of the bio-signal activity.Good spatial resolution may allow for more precise interpretation of theelectrical activity in the brain as well as muscular activity in theface and head. This may allow for improved accuracy in estimating auser's cognitive or emotional state.

The wearable computing device 100 may be embodied, for example, as awearable headset worn on a user's head. The wearable computing devicemay include a computing device, or connect to a computing device (notshown), and may be configured to create a VR environment on the HMD 110and sound generator 1140 for presentation to a user; receive bio-signaldata of the user from sensors such as electrode 130, optical device 140,electrode 149, eye tracker 150, breath sensor 160, electrode 170,non-contact electrode 180, sensors 192, bio-signal sensor 3500,bio-signal sensor 3700, capacitive electrode 4300, at least one of thebio-signal sensors comprising a brainwave sensor, and the receivedbio-signal data comprising at least brainwave data of the user; anddetermine brain state response elicited by the VR environment at leastpartly by determining a correspondence between the brainwave data and apredefined bio-signal measurement stored in a user profile, thepredefined bio-signal measurement associated with predefined brain stateresponse type. The brain state response may comprise an emotionalresponse type. The wearable device 100 may be in the form of a virtualreality headset.

In some embodiments, the wearable computing device 100 includes anelectronics module receiving bio-signals from sensors such as electrode130, optical device 140, electrode 149, eye tracker 150, breath sensor160, electrode 170, non-contact electrode 180, sensors 192, bio-signalsensor 3500, bio-signal sensor 3700, capacitive electrode 4300, or anycombination thereof. In some embodiments, the module includes analogsignal conditioning circuitry. In some embodiments, the electronicsmodule includes a processor. In some embodiments, the module includes awireless transmitter, such as a RF radio, for data transmission, or awired connection connecting to the HMD 110 and/or the computing device.In some embodiments, the electronics module is the computing device.

Embodiments of the wearable computing device 100 may provide for thecollection, analysis, and association of particular bio-signal andnon-bio-signal data with specific brain states for both individual usersand user groups. The collected data, analyzed data or functionality ofthe systems and methods may be shared with others, such as third partyapplications and other users. Connections between any of the computingdevices, internal sensors (contained within the wearable device),external sensors (contained outside the wearable device), usereffectors, and any servers may be encrypted. Collected and analyzed datamay be used to build a user profile that is specific to a user.

The user profile data may be analyzed, such as by machine learningalgorithms, either individually or in the aggregate to function as aBCI, or to improve the algorithms used in the analysis. Optionally, thedata, analyzed results, and functionality associated with the system canbe shared with third party applications and other organizations throughan API. One or more user effectors may also be provided at the wearabledevice or other local computing device for providing feedback to theuser, for example, to vibrate or provide some audio or visual indicationto assist the user in achieving a particular mental state, such as ameditative state.

In use, the device may detect whether a user noticed a transient ormoving stimulus in the visual or auditory field, and noticedcharacteristics of that stimulus encoded by the timecourse of thechange, and using that information detected by the transient EEGresponse. This can be used, for example, to signal to an outsideobserver (e.g. a clinician, researcher, or other person not in the sameVR environment) that the user has noticed or attended to the stimulus;to signal, via for example a change of facial expression on a virtual orholographic avatar, or a colour change of said avatar, to anotherobserver in the VR environment that the user noticed or attended to saidstimulus event; or to signal, via for example a change of facialexpressions of multiple avatars, or via an event in a VR environment,which of multiple users in said VR environment noticed or attended to astimulus event.

In some embodiments, the device may also detect a user's cognitive statebased on a combination of continuous brainwave signal and transientbrain responses to virtual stimulus events in the visual, auditory ortactile domain, in a VR environment, to predict thresholds for detectionof subsequent virtual events in the auditory, visual, or tactile field,and to optimize the presentation of subsequent stimuli in said VRenvironment for detection or to change the likelihood of the stimulusbeing either consciously attended or not consciously attended.

In some embodiments, the device may actively adapt the rate of stimuluspresentation based on a combination of continuous brainwave signal andtransient brain responses to virtual stimulus events in the visual,auditory or tactile domain, in a VR environment.

In some embodiments, the device may accept inputs from a head- orbody-worn continuous visual recognizer, such as a camera andcomputer/software system which recognizes objects, scenes, or actions inthe user's visual or auditory field, combines that information withbrainwave information time-synchronized to the visual field events via acomputer, and uses the combined information to determine whether theuser noticed the object, scene, or action, attended to the object,scene, or action, or whether the user recognized the object, scene, oraction.

In some embodiments, the system accepts inputs from a head- or body-worncontinuous auditory recognizer, such as a camera and computer/softwaresystem which recognizes objects, scenes, or actions in the user'sauditory or auditory field, combines that information with brainwaveinformation time-synchronized to the auditory field events via acomputer, and uses the combined information to determine whether theuser noticed the object, scene, or action, attended to the object,scene, or action, or whether the user recognized the object, scene, oraction.

In some embodiments, the system accepts inputs from a head- or body-worncontinuous visual recognizer, such as a camera and computer/softwaresystem which recognizes human faces in the user's visual or auditoryfield, combines that information with brainwave informationtime-synchronized to the visual field events via a computer, and usesthe combined information to determine whether the user recognized theface.

In some embodiments, the visual or auditory recognizer is not worn bythe user, but by another user, or is a stationary or object mountedrecognizer system. In some embodiments, electrodes on the face orforehead may measure muscle activity associated with facial expressionof emotions (for example: frown, surprise, puzzlement, sadness,happiness) in which the user's brainwaves are combined with bio-signalinformation about emotional facial expression to produce a change instate of a user's avatar in said VR environment.

In some embodiments, the diminution of a user's evoked brain response toa visual or auditory event in the VR environment (as in habituation orlearning) after repeated stimulus presentations may be used to predicthow frequently a new stimulus of a certain type should be presented tothe user to achieve familiarity—as in, for example, a memorization task,or a recognition task—and can be used to adapt an environment tooptimize engagement, or the retention of information.

In some embodiments, the diminution of a user's evoked brain response toa visual or auditory event in the VR environment (as in habituation orlearning) after repeated stimulus presentations may be used to predicthow frequently a new stimulus of a certain type should be presented tothe user to maintain a specific state of vigilance or responsiveness, orof interest. For example, a system designed to use brain responseinformation within a VR environment, which determines a user'slikelihood of loss of engagement or boredom, and adapts the environmentcontinuously to maximize engagement.

In some embodiments, a profile of the user including the user's brainresponse and engagement may be determined within the user's first fewminutes within a VR environment, and the environment is adapted to athreshold of interactivity to maintain engagement without continuouslymonitoring the user's brain response.

In some embodiments, multiple users in a VR environment, in which one ormore lead user (for example an instructor) is presented with informationoverlaid on another user's virtual space, or another user's avatar, mayallow the lead user to determine which of the other users (for examplestudents) attended to or were engaged with specific aspects ofinformation presented (for example, lesson elements) in the VRenvironment, based on the other users' individual brain responsessynchronized to the presentation of said information events. Forexample, in a virtual classroom, or in a physical classroom with mixedreality, a virtual display or information about what taught materialeach student is likely to have retained.

In some embodiments, the content is presented in the physicalenvironment.

User State Visualization

As an illustrative example, the VR environment may present informationabout a user's state. The wearable device processes user bio-signal dataand provides feedback through at least one feedback module. Feedback inthe VR environment may provide a more intuitive understanding for theuser's state than a regular display.

In one example, and having reference to FIG. 31, the user statevisualization includes a memory trace 3100. In some embodiments, atleast one object 3102 is created and projected in the VR environment.For example, as a user moves in a first direction, the user may leave atleast one object indicative of the user state. Alternatively, the atleast one object indicative of the user state is projected from a sourceand radiates in a propagation direction. For example, the at least oneobject 3102 is a continuous trail or a series of discrete objectsindicative of the user state. The at least one object 3102 is like abread crumb trail of the user state. For example, as shown in FIG. 31,objects 3102 are represented at current time t_(H), at time t_(H-1), attime t_(H-2), and at time t_(H-3). In this manner, the feeling of timeis made accessible to the user. In some embodiments, the origin of thetrace moves at a velocity in the first direction, V_(b), based on themovement of the user, or a manual user input, such as a controller 3104or other device, represented in the VR environment. In some embodiments,the elements of visual stimulus within a section of the trace propagatein a second direction at a velocity V_(m), such as being represented asa standing wave or moving sparkles. In some embodiments, for example,the trace represents linear time, and the elements moving at V_(m)represent a mental state within linear time. In changing the rate atwhich the at least one object is generated, the perception of time canbe altered. In some embodiments, user state can be used to affect therate at which the last one object is projected in the VR environment. Inthis manner, a feedback loop may be created that can help a user enterdifferent states of consciousness.

In another example, and having reference to FIG. 32, the user statevisualization includes a breath envelope 3200. In some embodiments, thebreath envelope is presented as a field surrounding the user's body 3202in the VR environment. The density of the field decays as distance, forexample, as indicated by x_(b) in the x-axis in FIG. 32, increases fromthe user. The size and density of the field is affected by the state ofthe user's breathing (such as breathing rate or a duration of a breath)and body oxygen content. In some embodiments, the bio-signal sensor usedto detect breathing is a breath sensor, for example, breath sensor 160.In some embodiments, breathing is detected using a stretchable strapworn on a user's chest (optionally including ECG or PPG functionality),accelerometers, gyroscopic sensors, or a combination thereof. In someembodiments, the bio-signal sensor used to detect the body oxygencontent includes a pulse oximetry sensor.

In some embodiments, the field is larger when the user has more air intheir lungs. In some embodiments, the field is more dense when the userhas more oxygen in their body. The field is used to affect a parameterof an object within the user's field to create a feeling of connectionbetween the breath and the user's immediate vicinity in the VRenvironment. In some embodiments, a denser field has a stronger effecton the parameter of the object. In some embodiments, the parameter is adimension or other behavior of the object. For example, if the userbreathes heavily for a period of time and increases the amount of oxygenin the body, the field can be used to distort the shape of objects, suchas making them larger within the acting range of the breath envelope. Insome embodiments, the more oxygenated the user becomes, objects withinthe breath envelope would be enlarged in the VR environment. Similarly,in some embodiments, sound may also be affected in a similar way. Anobject having a sound associated therewith may be modulated such that asound emitted by the sound generator and associated with the object maybecome louder or softer or change in spectral distribution within thebreath envelope field. In some embodiments, the breath envelope isdisplayed in the VR environment as an object.

In another example, and having reference to FIG. 33, the user statevisualization includes a heartwave manifold 3300. In some embodiments,the heartwave manifold 3300 is a field that radiates outward from theuser's virtual heart 3302. In some embodiments, the heartwave manifold3300 is synchronized with the user's heart. In some embodiments, theheartwave manifold 3300 is shaped as a sphere or ellipsoid. In someembodiments, the field includes at least one object 3304 in the VRenvironment directly visible to the user. In some embodiments, the atleast one object 3304 includes a series of spheres, shown in part inFIG. 33 as T_(H), T_(H-1) and T_(H-2), growing and propagating outwardradially, for example, at velocities V_(H), V_(H-1) and V_(H-2) as shownin FIG. 33, with the passage of time. In some embodiments, a new objectis created on each heartbeat. In some embodiments, each new objectpropagates away from the origin point. In such manner, the objects forma 3D ripple according to the heartbeats of the user. In someembodiments, the field is not directly visible in the VR environment. Insome embodiments, the field is interactive with the VR environment,giving the user indirect feedback from the user's heart. In someembodiments where the field is directly visible in the VR environment,the heartwave manifold includes information of the user's mind/bodystate associated therewith, allowing it to display the user's stateinformation recorded at that instant of time or vary in accordance withthe ongoing variation in the mind/body state. These patterns displayedon the heartwaves manifolds may change as they propagate outward in theVR environment.

In some embodiments where the field is directly visible in the VRenvironment, the user's brain state is rendered onto the at least oneobject of the heartwave manifold. In some embodiments, as the at leastone object of the heartwave manifold expands, the rendered spatialpattern may be associated with the depth in the brain from where theactivity is associated. For example, when one object of the at least oneobject of the heartwave manifold is first created, the user sees deepbrain activity. As the object propagates outward, the user sees activityat shallower depths of the brain. Eventually, the user sees surfaceactivity of the brain. In some embodiments, information could beintegrated from various sensors to represent a standard brain model, orcustomized to a user's fMRI-based brain model. In some embodiments, therendered spatial pattern is associated with the position, fromfront-to-back, of the brain, starting with the front of the frontal lobeto the rear of the occipital lobe. In some embodiments, the renderedspatial pattern is associated with the position, from starting from amidline and moves outward. In some embodiments, the rendered spatialpattern is associated with activity at different frequencies of a user'sbrainwave state. For example, when one object of the at least one objectof the heartwave manifold is first created, the user sees their brain'stheta wave activity. As the object propagates outward, the user sees theactivity of their brain at higher frequencies. Eventually, the user seestheir brain's gamma wave activity.

In some embodiments, the expansion of the at least one object is basedon time such that the mind/body state is less defined or visible as theat least one object expands outward. In some embodiments, thesignificance of the user's state at the time the at least one object wasgenerated affects the decay rate. For example, heartwave associated witha surprising event or strong emotional state may be visible for longer.Such surprising event or emotional state may be associated with an ERP,heart rate variability, skin galvanometry, anomalous movements (such asjerks, jumps, or microexpressions), or a combination thereof.

In some embodiments, the at least one object surface may be dynamicallyrendered based on the user's mind/body state in real time so that therendered surface reflects the user's current mind/body state. This mayfacilitate connection with other users in the VR environment. Theseusers may be physically proximate, or remote from one another. Inembodiments where the users are remote from one another, the users maybe connected to one or more computers for processing information in theVR environment via a computer network.

In some embodiments, information associated with the heartwave maychange over time as more time synchronized information becomesavailable. For example, if considering the action of a first user'sheart related to second user, the feedback loop is slowed by computerand communication lag, as well as brain associated perceptual lag.Information of the second user's heart reacting to the first user'sheart would be available at some later time than the first user'sheartbeat. When this information becomes available, the dynamic textureassociated with the at least one object of the heartwave manifold of thefirst user would change to reveal the relationship. This exemplaryinteraction allows the users to see heart based connection between them,as well as the transition between heart reactivity to heart coherence.

For example, ECG data and heart sensor data may be processed anddisplayed in the VR environment such that a heartbeat is presented as asphere. A property of the sphere, such as the size or color, may bemodified as the heartbeat changes. In some embodiments, the size may bedependent on the heart rate. For example, as a user's heart rateincreases, the sphere can grow in size, or go from a resting state color(e.g. green) to a exertion state color (e.g. red). Such integratedinformation may be more accurate in estimating a user's state thancontinuous EEG alone.

In another example, and having reference to FIG. 34, the user statevisualization includes an objectification field associated with thelevel of connection a user has with an object or another user in theenvironment. The objectification field may be displayed in the VRenvironment, showing the connectivity to the user or other users, or itmay be used to affect the environment or other users (such as throughvisual or tactile feedback provided to the other user) who are within orproximal to the field. For example, where an object is a non-playercharacter (“NPC”), the position of the eyes of the NPC may be modifiedin accordance with the objectification field. For example, the NPC'sgaze is modified to be aligned with a local objectification field. Thefield strength is directional and is associated with the user's interestlevel in another actor. In some embodiments, the user's interest inanother actor is determined based on ERP. The user's brain is determinedto be responsive to events involving the other user, relative to thatuser's baseline responsiveness to novel and familiar stimuli. As such,the objectification field is not always equal between two users. Forexample, user A's interest in user C, λ_(AC), may be different than userC's interest in user A, λ_(CA). In the case of an object B, the field ofuser A's interest in user B, λ_(AB), may be simple, for example decayingwith distance or visibility. For complex relationships, such as betweentwo users A and C, λ_(AC) and λ_(CA) can be used to calculate mutualinterest. For example, the mutual interest is the product of λ_(AC) andλ_(CA). The mutual interest optionally includes a coherence termindicating that the users are on the same wavelength. In someembodiments, the coherence term is computed using the level of synchronybetween the two users' time varying state feature vectors. For example,one type of synchrony includes the spectral coherence between two users'brainwaves who are proximal to each other in the VR environment.Synchrony between a user and an NPC or other non-user object is alsopossible. For example, this may be computed based on variance of thetime varying distance between the location of the user and the NPC orthe non-user object in the VR environment. In some embodiments, the VRenvironment is a game where a user dances with another user or an NPCand scoring is based on the synchrony of their movements.

User State Painting

In another exemplary application of the wearable computing device, theVR environment is a 3D painting application. The 3D painting applicationmay be similar to Tilt Brush™ from Google. In Tilt Brush™, a user (i.e.artist) is able to paint in multiple dimensions according to thepositioning of a controller. A brush stroke applied is an object in theVR environment. By incorporating the artist's brainwave state, thecolors may change based on the brain state of the individual. Thebrainwave state can dynamically determine the color that the brush willoutput. In this manner, a parameter of object (e.g. the color of aparticular portion of a brush stroke) in the VR environment isdynamically altered depending on the brainwave state of the user duringthe creation of the object. Alternatively, a brush stroke applied by auser may constantly change depending on the brainwave state of the user.In this manner, a parameter of object (e.g. the color of a particularportion of a brush stroke) in the VR environment is dynamically altereddepending on the brainwave state of the user after the creation of theobject. By modifying the color according to the brainwave state of theuser, a more direct emotional response can be output onto the virtualcanvas. In contrast, in a traditional, physical medium, the color of thebrush cannot be adjusted dynamically. By the time the artist mixes acolor according to their emotional state, the artist's emotional statemay have shifted and a new color may need to be mixed in order toreflect the new state. Further, the paint applied by the brush is notdynamically altered during a single stroke; in order to change the coloron the canvas, the new color of paint must be applied to the brush and anew stroke begun.

Meditation

In another exemplary application of the wearable computing device, theVR environment is a meditation application. While meditation can befelt, it can be hard to quantify in a way that other people canunderstand. It can also be easy for people doing meditation to feel asthough they have slipped behind or are not making enough progress. Thiscreates a distraction that is anathema to the act of meditation itself.Embodiments described herein may allow people who are meditating to seetheir progress as feedback. This can be helpful for people who have beenasked to meditate as part of cognitive behavioural therapy, or to bringdown blood pressure, or manage chronic pain.

For example, the user may want to participate in a meditation program ata crowded/noisy/non-conducive setting. To overcome this environmentalobstacle, the user dons a pair of VR goggles. The goggles provide avirtual meditation environment in which the area surrounding the user isfree of distractions. This VR environment can be mapped using a devicewhich contains sensors for mapping a 3D environment. The user canparticipate in either a walking or a sitting meditation practice withthe distracting elements of the setting blocked out. As the userpractices meditation, their EEG state is being monitored. The user canvisualize their EEG state during the meditation practice; it can bepresented like a music visualizer—a series of peaks and troughs canbecome visible travelling towards them, corresponding to their mentalstate. Alternatively, their EEG state can modify the VR environmentitself. For example, the 3D environment can be a beach with the EEGstate being represented visually and/or aurally by waves washing to theshore. The VR environment may be further modified by other user statedata. For example, a user's heart rate can be represented by the cloudsin the sky. This can allow the user to see and hear how their meditationis progressing, i.e. whether they are meeting their meditation goals interms of relaxation, etc. The user can then optimize their meditationpractice to meet specific goals by modifying their breathing or someother variables to create a different outcome during the meditationpractice.

Embodiments described herein translates EEG data, heart rate and pulsedetection, and eye-tracking to generate feedback outputs like real timedynamic changes to the VR environment (such as a change in music orambient sound, or shifts in light, transparency, or opacity, ortopography), while also creating data for reports that wearers could optinto or share with friends and supporters.

According to an aspect, there is provided a system for detecting auser's notice to a transient or moving stimulus in the user's visual orauditory field in a virtual or mixed environment, and to characteristicsof that stimulus encoded by the timecourse of the change, and using theinformation detected by a transient EEG response for: signalling to anoutside observer (a clinician, researcher, or other person not in thesame virtual or mixed reality environment) that the user has noticed orattended to the stimulus; signalling, to another observer in the virtualor mixed environment, that the user noticed or attended to said stimulusevent; or signalling, via for example a change of facial expressions ofmultiple avatars, or via an event in a virtual environment, which ofmultiple users in said virtual or mixed reality environment noticed orattended to a stimulus event.

In some embodiments, the signalling to another observer is effected viaa change of facial expression on a virtual or holographic avatar, or acolour change of said avatar.

According to an aspect, there is provided a system for detecting auser's cognitive state based on a combination of continuous brainwavesignal and transient brain responses to virtual stimulus events in avisual, auditory or tactile domain, in a virtual, augmented or mixedreality environment, to predict thresholds for the user's detection ofsubsequent virtual events in the auditory, visual, or tactile field, andto optimize the presentation of subsequent stimuli in said virtual,augmented or mixed reality environment for detection or to change thelikelihood of the stimulus being either consciously attended or notconsciously attended.

According to an aspect, there is provided a system for actively adaptinga rate of stimulus presentation based on a combination of continuousbrainwave signal and transient brain responses to virtual stimulusevents in the visual, auditory or tactile domain, in a virtual,augmented or mixed reality environment.

According to an aspect, there is provided a system which accepts inputsfrom a head- or body-worn continuous visual recognizer, such as a cameraand computer/software system which recognizes objects, scenes, oractions in the user's visual or auditory field, combines thatinformation with brainwave information time-synchronized to the visualfield events via a computer, and uses the combined information todetermine whether the user noticed the object, scene, or action,attended to the object, scene, or action, or whether the user recognizedthe object, scene, or action.

According to an aspect, there is provided a system which accepts inputsfrom a head- or body-worn continuous auditory recognizer, such as acamera and computer/software system which recognizes objects, scenes, oractions in the user's auditory or auditory field, combines thatinformation with brainwave information time-synchronized to the auditoryfield events via a computer, and uses the combined information todetermine whether the user noticed the object, scene, or action,attended to the object, scene, or action, or whether the user recognizedthe object, scene, or action.

According to an aspect, there is provided a system which accepts inputsfrom a head- or body-worn continuous visual recognizer, such as a cameraand computer/software system which recognizes human faces in the user'svisual or auditory field, combines that information with brainwaveinformation time-synchronized to the visual field events via a computer,and uses the combined information to determine whether the userrecognized the face.

In some embodiments, the visual or auditory recognizer is not worn bythe user but worn by another person, or being a stationary or objectmounted recognizer system.

In some embodiments, the system further comprises an additional input ofelectrodes on the face or forehead to measure muscle activity associatedwith facial expression of emotion in which the user's brainwaves arecombined with bio-signal information about emotional facial expressionto produce a change in state of a user's avatar in said virtualenvironment.

In some embodiments, the emotion includes frown, surprise, puzzlement,sadness, or happiness.

In some embodiments, the diminution of a user's evoked brain response toa visual or auditory event in the virtual environment (as in habituationor learning) after repeated stimulus presentations is used to predicthow frequently a new stimulus of a certain type should be presented tothe user to achieve familiarity.

In some embodiments, the new stimulus includes a memorization task, or arecognition task.

In some embodiments, the diminution of the user's evoked brain responseis used to adapt an environment to optimize engagement, or the retentionof information.

In some embodiments, the diminution of a user's evoked brain response toa visual or auditory event in the virtual environment (as in habituationor learning) after repeated stimulus presentations is used to predicthow frequently a new stimulus of a certain type should be presented tothe user to maintain a specific state of vigilance or responsiveness, orof interest.

In some embodiments, the system uses brain response information within avirtual or mixed reality environment, which determines a user'slikelihood of loss of engagement or boredom, and adapts the environmentcontinuously to maximize engagement.

In some embodiments, a user's brain response and engagement isdetermined quickly, within the user's first few minutes within thevirtual or mixed reality environment, and the environment is adapted toa set point level of richness to maintain optimal engagement withoutcontinuously monitoring the user's brain response.

In some embodiments, there are multiple users in a virtual or mixedreality environment, in which one or more lead user (for example aninstructor) is presented with information overlaid on a user's virtualspace, or a user's avatar, to allow the lead user to determine whichother users (for example students) attended to or were engaged withspecific aspects of information presented (for example, lesson elements)in the virtual or mixed reality environment, based on the other users'individual brain responses synchronized to the presentation of saidinformation events.

In some embodiments, the environment is a virtual classroom, or aphysical classroom with mixed reality, a virtual display or informationabout what taught material each student is likely to have retained.

In some embodiments, content is presented in the physical environment.

According to an aspect, there is provided a system, apparatus, device,process, or method including one or more features as set out in thedescription, claims, drawings, or any combination thereof.

General

It will be appreciated that any module or component exemplified hereinthat executes instructions may include or otherwise have access tocomputer readable media such as storage media, computer storage media,or data storage devices (removable and/or non-removable) such as, forexample, magnetic disks, optical disks, tape, and other forms ofcomputer readable media. Computer storage media may include volatile andnon-volatile, removable and non-removable media implemented in anymethod or technology for storage of information, such as computerreadable instructions, data structures, program modules, or other data.Examples of computer storage media include RAM, ROM, EEPROM, flashmemory or other memory technology, CD-ROM, digital versatile disks(DVD), blue-ray disks, or other optical storage, magnetic cassettes,magnetic tape, magnetic disk storage or other magnetic storage devices,or any other medium which can be used to store the desired informationand which can be accessed by an application, module, or both. Any suchcomputer storage media may be part of the mobile device, trackingmodule, object tracking application, etc., or accessible or connectablethereto. Any application or module herein described may be implementedusing computer readable/executable instructions that may be stored orotherwise held by such computer readable media.

Thus, alterations, modifications and variations can be effected to theparticular embodiments by those of skill in the art without departingfrom the scope of this disclosure, which is defined solely by the claimsappended hereto.

In further aspects, the disclosure provides systems, devices, methods,and computer programming products, including non-transientmachine-readable instruction sets, for use in implementing such methodsand enabling the functionality described previously.

Although the disclosure has been described and illustrated in exemplaryforms with a certain degree of particularity, it is noted that thedescription and illustrations have been made by way of example only.Numerous changes in the details of construction and combination andarrangement of parts and steps may be made. Accordingly, such changesare intended to be included in the invention, the scope of which isdefined by the claims.

Except to the extent explicitly stated or inherent within the processesdescribed, including any optional steps or components thereof, norequired order, sequence, or combination is intended or implied. As willbe will be understood by those skilled in the relevant arts, withrespect to both processes and any systems, devices, etc., describedherein, a wide range of variations is possible, and even advantageous,in various circumstances, without departing from the scope of theinvention, which is to be limited only by the claims.

What is claimed is:
 1. A mediated reality apparatus comprising: awearable computing device comprising a bio-signal sensor to receivebio-signal data from a user, a display to provide an interactivemediated reality environment for the user, a display isolator, thebio-signal sensor comprising a brainwave sensor, wherein the bio-signalsensor is embedded in the display isolator, the bio-signal sensor havinga soft, deformable user-contacting surface with a bio-signal sensor, toreceive bio-signal data from a user; the computing device incommunication with a processor configured to: as part of the interactivemediated reality environment, present content via the display, thecontent including an object in the mediated reality environment; receivethe bio-signal data of the user from the bio-signal sensor; process thebio-signal data to determine user states of the user, including brainstates, the user states processed using a user profile stored in a datastorage device accessible by the processor and the user states includingbrain states; modify a parameter of the object in the interactivemediated reality environment in response to the user states of the user,wherein the user receives feedback indicating the modification of theobject via the display.
 2. The apparatus of claim 1 wherein theprocessor is configured to: detect the user's interest in the object,and modify the parameter of the object in response to the user'sinterest.
 3. The apparatus of claim 1, wherein the wearable computingdevice comprises a display isolator, wherein the bio-signal sensor isembedded in the display isolator, wherein the bio-signal sensor has asoft, deformable user-contacting surface.
 4. The apparatus of claim 1,wherein the processor is configured to: connect with a remote feedbackdevice for presenting an indication of the user's interest to anobserver.
 5. The apparatus of claim 1, wherein the processor isconfigured to: create and/or modify another object in the mediatedreality environment in response to the user's interest in the object. 6.The apparatus of claim 1, wherein the user profile includes a thresholdfor detection of a virtual event presented to the user by the displaydetermined using the bio-signal data obtained concurrently with previousvirtual events presented to the user.
 7. The apparatus of claim 6,wherein the threshold for detection is modified based on the bio-signaldata obtained during the presentation of the virtual event.
 8. Theapparatus of claim 6, wherein the processor is configured to modify thecontent being presented in the mediated reality environment based on thethreshold for detection for optimizing user engagement.
 9. The apparatusof claim 1, wherein the mediated reality apparatus includes a trackerfor detecting the user's physical environment and the processor isconfigured to modify the content of the mediated reality environmentbased on properties of the physical environment.
 10. The apparatus ofclaim 1, wherein the processor communicates with effectors in the user'sphysical environment for modifying the physical environment.
 11. Acomputer-implemented method comprising: receiving, from a bio-signalsensor, bio-signal data of a first user of multiple users in a virtualor mixed environment, the bio-signal sensor integrated as part of awearable computing device having at least one feedback module to providethe virtual or mixed environment for the first user, the bio-signalsensor comprising a brainwave sensor. determining a transientelectroencephalogram response of the first user, based on at least thebio-signal data; detecting, based at least in part on the transientelectroencephalogram response, the user's notice or attendance to achange in a transient or moving stimulus in the user's visual orauditory field in the virtual or mixed environment, and ofcharacteristics of that stimulus encoded by the timecourse of thechange; signalling to an outside observer that the user noticed orattended to the stimulus; signalling, to a second user in the virtual ormixed environment, that the first user noticed or attended to thestimulus; and signalling, via an event in the virtual or mixedenvironment, which of the multiple users in said virtual or mixedreality environment noticed or attended to the stimulus.
 12. The methodof claim 11, wherein the signalling to the second user is effected via achange of facial expression on a virtual or holographic avatar, or acolour change of said avatar.
 13. The method of claim 11, furthercomprising measuring, using input of electrodes on the user's face orforehead, muscle activity associated with a facial expression ofemotion; combining the user's brainwaves with bio-signal informationabout the facial expression; and producing a change in state of theuser's avatar in said virtual or mixed environment based at least inpart on the combined user's brainwaves and bio-signal information. 14.The method of claim 1, further comprising: detecting diminution of thefirst user's evoked brain response to a visual or auditory event in thevirtual or environment after repeated stimulus presentations to predicthow frequently a new stimulus of a certain type should be presented tothe user to achieve familiarity.
 15. A mediated reality devicecomprising: a wearable computing device with a bio-signal sensor, atleast one feedback module to provide an interactive mediated realityenvironment for a user, the bio-signal sensor receives bio-signal datafrom the user, the bio-signal sensor comprising a brainwave sensor,wherein the bio-signal sensor comprises: a body, an electrode extendableinto the body, the electrode having a contact end configured to receivean electrical bio-signal from a user's skin, wherein in response to adownward force acting on the bio-signal sensor to urge the bio-signalsensor against the user's skin and upon contact with the user's skin,the electrode is configured for movement into the body along a movementaxis, an actuator attached to the body and operatively connected to theelectrode urging the electrode out of the body along the movement axistoward an extended position, wherein in the absence of the downwardforce, the electrode is disposed in the extended position, and a contactadjuster connected to the electrode, the contact adjuster including ahandle manipulatable by the user to reduce noise the electricalbio-signal caused by impedance of the user's hair.
 16. The device ofclaim 15, wherein the contact adjuster is configured to rotate theelectrode along a plane that is substantially perpendicular to themovement axis.
 17. The device of claim 15, wherein the actuator includesa coil spring fixed on one end to the body and biased against theelectrode on the other end, and wherein the contact adjuster includes ashaft extending through the compressive axis of the coil spring fortranslating rotational forces perpendicular to the movement directionfrom the handle to the electrode, translational forces along themovement direction from the handle to the electrode, or both.
 18. Thedevice of claim 15, further comprising a rotational limiter for limitingthe rotational movement of the electrode.
 19. The device of claim 15,wherein the contact end of the electrode includes a collection plate anda plurality of prongs extending from the collection plate, wherein eachprong includes a distal tip for contacting the user's skin.
 20. Thedevice of claim 15, wherein the body includes a contact end, wherein thecontact end includes at least one groove for receiving at least aportion of the user's hair therein.